FortiGate Link Monitor Failover Issue

[kumarmt]

I'm experiencing intermittent issues with link monitor failover on my FortiGate device. I have four WAN connections (internal7, wan1, dmz, and wan2) set up with the following link monitor configuration:

config system link-monitor
    edit "1" (internal7)
        set srcintf "internal7"
        set server "8.8.8.8"
        set protocol ping
        set gateway-ip 0.0.0.0
        set source-ip 0.0.0.0
        set interval 5
        set timeout 1
        set failtime 5
        set recoverytime 5
        set ha-priority 1
        set update-cascade-interface enable
        set update-static-route enable
        set status enable
    next
    edit "wan1"
        set srcintf "wan1"
        set server "8.8.8.8"
        set protocol ping
        set gateway-ip 0.0.0.0
        set source-ip 0.0.0.0
        set interval 5
        set timeout 1
        set failtime 5
        set recoverytime 5
        set ha-priority 2
        set update-cascade-interface enable
        set update-static-route enable
        set status enable
    next
    edit "dmz"
        set srcintf "dmz"
        set server "8.8.8.8"
        set protocol ping
        set gateway-ip 0.0.0.0
        set source-ip 0.0.0.0
        set interval 5
        set timeout 1
        set failtime 5
        set recoverytime 5
        set ha-priority 3
        set update-cascade-interface enable
        set update-static-route enable
        set status enable
    next
    edit "wan2"
        set srcintf "wan2"
        set server "8.8.8.8"
        set protocol ping
        set gateway-ip 0.0.0.0
        set source-ip 0.0.0.0
        set interval 5
        set timeout 1
        set failtime 5
        set recoverytime 5
        set ha-priority 4
        set update-cascade-interface enable
        set update-static-route enable
        set status enable
    next
end

The issue is that after a WAN connection, such as internal7, goes down and then recovers, the FortiGate doesn't consistently revert back to it. This behavior occurs intermittently, and the link monitor doesn't always trigger the expected failover.

I've already verified the configurations, monitored the logs, and tested different scenarios, but the issue persists. I'm looking for guidance on how to ensure that the FortiGate reliably reverts to the primary connection (internal7) once it has recovered.
The device i am testing is with fortigate 60D and 80F

Any insights or suggestions on improving the configuration or troubleshooting this issue would be greatly appreciated. Thank you!

[xshkurti · Accepted solution]

@kumarmt 
You can use SD-WAN manual rule creation, and select only unlimited internet interfaces.

They will be used everytime traffic leaves fortigate.
When they fail, the last rule will be used, and the last rule includes all 4 interfaces, but since 2 unlimited are failing, only pay-as-you-use will be used. So this is a scenario where you have backup (failover) and not load-balance.

So yes you can do it via sdwan.

More info:
Technical Tip: Assigning Priority to SD WAN Member... - Fortinet Community

Regards,

[ezhupa]

Hello kumarmt, 

Your link-monitor configuration seems ok. Basically what the link monitor will do is remove the route when it is dead and reinstall the route as soon as the link monitor is alive according to the values configured in it. 
Depending on your routing setup, if the route through internal7 has the lowest AD when it comes back to the routing table it should be the preferred route and the connection would switch to that port.  I believe what could affect routing in this case would be 2 settings: 
1) set preserve-session-route enable/disable 
https://community.fortinet.com/t5/FortiGate/Technical-Tip-Enabling-the-preserve-session-route/ta-p/1...
2) set snat-route-change [disable|enable] In case NAT is being performed. 

config system global
     set snat-route-change [disable|enable]
   end

https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-Routing-Changes-and-SNAT-snat-route-...

[kumarmt]

@ezhupa 

Thank you for your previous assistance.

I have attempted to enable Source Network Address Translation (SNAT) and preserve-session. The failover process operates as expected; however, the failback to the lowest Active Directory (AD) is not functioning correctly. This issue seems to arise when utilizing 4 WAN connections, as opposed to the expected behavior with only 2 WAN (my expirence).

In addition, I am considering implementing Software-Defined Wide Area Networking (SD-WAN) following the guidance provided by @xshkurti . I will keep you informed of the outcomes.

Appreciate your ongoing support.

[xshkurti]

@kumarmt 

I would also suggest you to change your configuration and use SD-WAN instead of link-monitor and static routers.

This way SD-WAN will use its monitoring design to update static routes and steer traffic to the proper interface.
SD-WAN uses another similar mechanism which is called performance SLA for link health checks (similar to link-monitor) but build in this feature.

You should consider changing to this solution.

More info about SD-WAN:
What is Secure SD-WAN? How Does it Work? | Fortinet
Configuring the SD-WAN interface | FortiGate / FortiOS 7.4.1 | Fortinet Document Library
Technical Tip: How to load balance traffic betwee... - Fortinet Community
Technical Tip: SDWAN Performance SLA with Multiple... - Fortinet Community

Regards,

[kumarmt]

@xshkurti Thank you for your feedback.
I am trying to implementing this on Marine industry. 
Using internet as (Starlink(very fast internet), VSAT Unlimited data usage(Slow internet), VSAT Pay-as-you-use(Slow internet),FBB Pay-as-you-use(very slow internet))

"I have aslo planned implementation of SD-WAN. In my scenario, we have a combination of total four internet connections, where two are pay-as-you-use and the other two are unlimited data plans. However, the pay-as-you-use lines are relatively expensive.

My goal is to use the pay-as-you-use internet connections as a failover option only when the two unlimited data lines are unavailable. Is it possible to configure SD-WAN to achieve this specific failover behavior?"

[xshkurti · Accepted solution]

@kumarmt 
You can use SD-WAN manual rule creation, and select only unlimited internet interfaces.

They will be used everytime traffic leaves fortigate.
When they fail, the last rule will be used, and the last rule includes all 4 interfaces, but since 2 unlimited are failing, only pay-as-you-use will be used. So this is a scenario where you have backup (failover) and not load-balance.

So yes you can do it via sdwan.

More info:
Technical Tip: Assigning Priority to SD WAN Member... - Fortinet Community

Regards,

[xshkurti · Accepted solution]

@kumarmt 

In your case, i have created a test environment to show you how your config should look.
First create sdwan zone and put all 4 internet links

Second go to SD-WAN Rules and create a rule that should look like this:

So in SD-Wan rules you will see something like this:

This means that for all traffic only 2 internet links will be used.
If they fail, the last rule that has all 4 members (but 2 are alredy down as we assume will happen with unlimited ISPs) only 2 Pay-As-You-Go will be hit.

Hope this is clear for implementation.

If you have found this as a solution, please like and accept it to make it easily accessible for others.

Regards!

[kumarmt]

Thank you very much. I now have a clearer understanding, and the failover is performing as expected. I appreciate your guidance on this matter.

[mle2802]

Hi @kumarmt,

In addition to ezhupa comment, what is your static route configuration? Is the route via internal7 has the lowest AD and priority?

[kumarmt]

@mle2802 
Yes internel7 has lowest distance

show full-configuration router static
config router static
    edit 2
        set status enable
        set dst 0.0.0.0 0.0.0.0
        set gateway 192.168.2.35
        set distance 50
        set weight 0
        set priority 50
        set device "wan2"
        set comment ''
        set blackhole disable
        set dynamic-gateway disable
        set virtual-wan-link disable
        set dstaddr ''
        unset internet-service
        set internet-service-custom ''
        set link-monitor-exempt disable
    next
    edit 3
        set status enable
        set dst 0.0.0.0 0.0.0.0
        set gateway 192.168.5.1
        set distance 20
        set weight 0
        set priority 20
        set device "wan1"
        set comment ''
        set blackhole disable
        set dynamic-gateway disable
        set virtual-wan-link disable
        set dstaddr ''
        unset internet-service
        set internet-service-custom ''
        set link-monitor-exempt disable
    next
    edit 4
        set status enable
        set dst 0.0.0.0 0.0.0.0
        set gateway 192.168.6.1
        set distance 30
        set weight 0
        set priority 30
        set device "dmz"
        set comment ''
        set blackhole disable
        set dynamic-gateway disable
        set virtual-wan-link disable
        set dstaddr ''
        unset internet-service
        set internet-service-custom ''
        set link-monitor-exempt disable
    next
    edit 5
        set status enable
        set dst 0.0.0.0 0.0.0.0
        set gateway 192.168.100.1
        set distance 10
        set weight 0
        set priority 10
        set device "internal7"
        set comment ''
        set blackhole disable
        set dynamic-gateway disable
        set virtual-wan-link disable
        set dstaddr ''
        unset internet-service
        set internet-service-custom ''
        set link-monitor-exempt disable
    next
end