Issue with SD-WAN

[tedew]

hello,

I have issue (maybe wrong configured) with my SDWAN.

We have one site with only one  Direct Internet connection (let's say SITE-R) and second link (as dark fiber) to Main Site (let's say SITE-M) and IPSec tunnel configured on this darkfiber.

And to access to the Interent for Users in Site-R I configured SDWAN rule as below ;

 First use Direct Internet connection in Site-R, if Direct connection link is not meet SLA or not working at all,  use IPSec to Site-M and use Internet from Site-M - lowest cost LSA.  (I have rule in Site-M on firewall which allow this kind of traffic)

I also configured SLA and included Direct Internet Access only in Site-R and i'm pinging 8.8.8.8.

But todya we had issue with this DIA in Site-R and Internet doesn't work.

Full pictuer of SD-WAN config

One Zone: External

Members: 3 - ISP, IPsec to Site-M over ISP, IPSec to Site-M over Darkfiber

Rules: 1. TO-Site-M (members both IPSec tunels) - Lowest cost SLA

           2. To-INTERNET (members: ISP, IPSec to Site-M over Darkfiber) - Lowest Cost LSA

SLA: 1 - using by To-INTERNET rule (ping 8.8.8.8) - only ISP is a member, update routing is selected

          2 - using by TO-SITE-M rule (ping device in Site-M) - both IPsec are members, update routing is deselected

How can I diagnose issue ??  I think that this config is ok, but maybe my idea is wrong :)

Thanks 

[funkylicious]

Hi,

First of all, did this setup ever worked as intended?

Secondly, in the routing table, I assume that you have 3 static default routes, one using the ISP link, the rest using the IPsec tunnels.

Can you show a output of the command, get router info routing-table all ?

[tedew]

hello,

Yes, when all is UP it's working, I mean Internet traffis is using ISP interface, and Internal traffic to Site-M is using IPSec over DarkFiber.

I can't share full routing tab, but i will try descreib (i will hide Public IP of ISPs):

I have in static routing configured on FGT:

0.0.0.0/0 - ISP

0.0.0.0/0 - IPSec to Site-M over DarkFiber

10.16.0.0/16 - IPSec to Site-M over Darkfiber

10.16.0.0/16 - IP Sec to Site-M over ISP

S* 0.0.0.0/0 [10/0] via ISP_IP, wan1, [1/0]
[10/0] via IPSEC-TO-Site-M-DF tunnel 10.0.254.1, [21/0]

S 10.16.0.0/16 [1/0] via IPSEC-TO-Site-M-DF tunnel 10.0.254.1, [1/0]
[1/0] via IPSEC-TO-Site-M-ISP tunnel Ip-ISP, [1/0]

[funkylicious]

I assume that on the IPsec tunnel you have 0.0.0.0/0 defined as local/remote selectors at both ends and in Site-M you have a route back via the to the local subnet of Site-R and firewall rules allowing internet access to that remote subnet in Site-M ?

If those from above are true, my best bet is that when the SLA fails on ISP link, it doesnt remove the route from the routing-table.

You can confirm this, in a maintenance window and shutdown the ISP link and check again the routing table and see if ISP link is removed from it.

If it's removed from RT then most likely there is an issue on the remote FGT in Site-M where you should do a debug of the traffic flow.

[tedew]

hello,

1 - 0.0.0.0/0  definded (local/remote) at both end of IPSec over Fiber

2. yes, I have static route on Site-M - route 10.17.0.0/16 over IPSec over DarkFiber to Site-R
(Site-M is using 10.16.0.0/16 and Site-R is using 10.17.0.0/16)

3-yes I have rule on Firewall in Site-M(remote 10.17.0.0/16) to all (Internet) 

One difference which is see is that:

On Site-R in static routing configuration I see SDWAN memebers Interfaces as destination Interface

On Site-M is static routing configuration I see SD WAN Zone as destination Interface