latency when trafic on unrelated interfaces

I have a setting with a (cluster of) 1801F, all internal networks are vlan interfaces is on a 2x10G LACP.

I have network A which does mostly webRTC to network B. When there is an important traffic between some other vlans (let's say C and D, but not always the same), latency between A and B grows, such that clients see the latency.

What should I look for in the configuration ?.

The policy for traffic from A to B has full logging, may it cause some latency ?

Fortigate 7.0.17