FortiGate introducing Latency in Packet Flow

Are you able to elaborate a bit more on 'ACI' in this context? 
What do you have enabled on  the policies for this traffic?
What is the latency from the client to the FortiGate, and from the FortiGate to the server?

Appologies, Can't draw it properly due to tight schedule. As we can see in the diagram below as soon as the trace lands on Fortigate response time jumpe to 35.398ms. 8th hope is FortiGate incoming Interface for this connection.  

Can you post the configuration of the particular firewall policy that matches this traffic ? 

Firewall Policy is allowing all with no security Controls applied 

config firewall policy
edit 6
set name "Primary>DR"
set uuid afcf94c4-6f4e-51ef-4a4b-866241348179
set srcintf "FW_Impct_L3_Out"
set dstintf "FW_Impact_L3_IN"
set action accept
set srcaddr "172.20.14.0/25" "172.20.10.128/25"
set dstaddr "172.24.5.0/25" "172.24.4.128/25"
set schedule "always"
set service "ALL"
set logtraffic all
next
end

Hello

Please collect packet capture (Wireshark) on Firewall and source and destination to see the traffic flow

  https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-Packet-Capture-on-FortiOS-GUI/ta-p/194444

I had a similar case about 3 months ago, when I upgraded my fortigate 60f from 7.2.4 to 7.2.7 version. avg. response time jumped from 1ms to 50ms (in local network). Even fortinet support had trouble finding what was an issue, we tried firmware upgrade as our last resort. Upgrade solved the issue but we couldn't conclude what was the problem.