80d Loses routes for VOIP devices?

I've this issue with one of my units in that it will randomly lose routes for some VOIP phones.

Some context:

Site A (main) - VOIP controller exists

Site B (remote) - VOIP phones need to connect via IPSEC tunnel to Site A controller (static IP).

Site A - B have had a tunnel for years, this has worked fine.

When this happens I've run a packet capture and instead of the routes going out the tunnel to main, they get "stuck" internally at Site B. However, pinging via my PC (or any PC) works fine. Rebooting the Fortinet immediately resolves the issue.

Any ideas why this is? I've a very similar setup at 2 other (different connections, same setup) locations and have never had this issue before. I've also tried to make a static route to the controller with a lower cost than the regular subnet, this had no impact.