Have you tried the following:
config system settings
set sip-helper disable
set default-voip-alg-mode kernel-helper based
Thank you, yes I tried all of those, spend 4 hours on a support call with Fortinet, we checked everything, and for the most part, we had literally every SIP/VOIP affecting service turned off. I then noticed 2 days ago, that our traffic shaping policy I had only applied the guaranteed bandwidth on the Shared Shaper, not on the Reverse Shaper. I have since applied a guaranteed shaper in both directions, as well as applied a per-IP max bandwidth that is more than enough for a VOIP call (Polycom hard phone), and the total amount of phones add up to less than my guaranteed setting. Since having done that change, it seems like my problem has stopped happening (also improved jitter on all calls from that location).
Nowhere does it recommend/suggest this in FortiNet's documentation, but it seems to have helped.