SD-Wan failover and failback

[LCIT]

I'm playing in Eve-NG, trying to get SD-WAN setup.  I've got everything configured but something doesn't seem to be correct.

I've got (within eve-ng) an 8.8.8.8 that I'm pinging as my performance SLA.  I've got some NETem devices between my FortiGate FFVMEV (v7.0.13) and the router at 8.8.8.8.  I've WAN1 and WAN2 both set up.  Using NETem I've set a 10ms (each way) delay on WAN2, so WAN1 should be the preferred route.

I go to Network - SD-WAN - SD-Wan Rules and I see that per my SD-Wan rule, WAN1 is the current route.  I ping from a device behind the FortiGate to 8.8.8.8 and traffic heads out WAN1.  Now I introduce some latency into WAN1.  I see in my SD-WAN, SD-WAN Rules that now my preferred route is WAN2, but my ping continues to go through WAN1 unless I forcefully clear the session in the FortiGate.  If I take away the latency in WAN1, again I see per the SD-WAN rules that WAN1 is the preferred route, but the ping continues to go out WAN2, unless I again forcefully clear the session in the FortiGate.  Alternatively if I stop the ping, wait for about a minute for the session to timeout in the FortiGate, then I don't have to forcefully clear the session in the FortiGate.  
If I manually kill a link in the eve-ng network from the fortigate to the 8.8.8.8 router, after a few missed pings, the connection fails over to WAN2 and I see in the SD-WAN rules that WAN2 is now my preferred route.  If I bring the link back online in the eve-ng network, I in the SD-WAN rules that WAN1 is now my preferred route again, but again, unless I forcefully kill the session traffic continues to go out WAN2.
It seems like unless that session clears, the SD-WAN rule isn't having any effect on traffic.  Is this correct?  It seems like if I've popped my SLA, the fortigate should be switching the traffic to the best route but it only seems to do this after the session has expired.

[Bjay_Prakash_Ghising]

You can follow the attached link. The issue has been highlighted here.

You can enable snat route change for the session to choose the best path.

config system global
    set snat-route-change enable

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Using-SNAT-route-change-to-update-existing...

HTH

Kind Regards, 

Bijay Prakash Ghising

[LCIT]

Yeah, I found that and tried that, same result.

[Bjay_Prakash_Ghising]

Do you have the preserve session route enabled on the interface?

Otherwise, it is expected to carry on with the newly switched ISP link. 

Also, can you share the SD-WAN rule(CLI and GUI) that the traffic is passing through? How did you configure it?

[LCIT]

Configured it via the GUI.  I don't know the CLI very well.  

[Bjay_Prakash_Ghising]

Can you confirm in the GUI that the SD-Rule hit count is increasing? As of now, all we can check is whether the traffic passing through the expected rule or not. 

Kind Regards, 

Bijay Prakash Ghising

[LCIT]

Yes, it's increasing and as I stated, if I let the session clear (either naturally or force the session to clear) the traffic flows out the correct interface.

[LCIT]

It took me a bit to find preserve session route, assuming that's on the WAN interface and not somewhere within the SD-WAN configuration, it is disabled.

[LCIT]

[LCIT]

So this is interesting.  While I tried the 
config system global
    set snat-route-change enable
and it didn't work, I thought "what if somehow, the command just wasn't taking effect".  So I enabled the command, then rebooted the FortiGate.  Now things work as expected!  Very unusual to have to reboot a FrotiGate to get it to work right.  Odd as heck!