IPSEC tunnel status up but no traffic, Have to restart tunnel for traffic to come back up

Hi I'm running IPsec tunnel between two fortigates. 

The tunnel status shows up, but once in a while I can't ping the other end until After I restart the tunnel.

 I have autokey-keep alive and auto-negotiate checked on both sides.

Anybody know what's up with this?