Replay detection?

Not sure what your  questions are. But I have never to mess with replay protection to make a vpn stable. It should be on by default with latest fortiOS versions fwiw.

1st you need to understand what replay window does. It's a means to monitor X amount of ESP sequence from the highest received seq#. So what that means, you have a window that's set and if a previous or older sequence is received that's lower than the high sequence,  replay protection kicks in. It controls and protects from already processed ESP datagrams. 

I'm only seen the above becoming an issues if you have poor infrastructure, poor/weird ECMP and with poor infrastructure , or high queuing when COS/QOS that drain high-buckets before lower buckets where packets could be delayed.

So in your case, do you see icmp-responses being duplicated between ike-gateway ? High packet lost ? traceroutes that shows weird ECMP along the path? You said pings works ( guessing  ) between  local+remote subnets, do you see packet lost or out or order or duplicate pings aka DUP! ?

Have you double check the ph1/ph2 settings on local and remote gateways? What device are you connecting the fgt to ? Have you ran "diag debug flow" ? What is the replaywindow size ? ( should be 2048  it used to be 1024 in older fortios versions...I do not believe you can even set the window size  it's either enable or disable ) 

Ken Felix Security Blog: ESP replay window enabling & disable Fortigate (socpuppet.blogspot.com)

e.g

# cli cmd 

  diag vpn tunnel list | grep replaywin

And lastly, have you check the path-mtu? You might need to adjust the TCP traffic path-mtu.

# cli cmd

    diag vpn tunnel list | grep mtu

I would determine the path-mtu and ensure that traffic ( syn and syn-ack  tcp.mss ) is clamp to the path mtu or smaller  than the above listed output.

# cli cmd  to set syn/syn-ack size

config firewall policy 

#

# policy number that controls the traffic

#

    edit <xxxxxxx>        

     set tcp-mss-sender1380      set tcp-mss-receiver 1380

 end

YMMV,  but window-replay is probably not your issue.

Ken Felix