Azure Active Active pair

I have an Azure instance of 2 Fortigates in active/active via fabric connector.  They have ipsec tunnels to 2 500D's at 2 different branch locations.  When both firewalls are up in Azure, the tunnels tend to drop packets significantly and I am seeing ESP errors on the 500D's "Received ESP packet with unknown SPI".  As soon as I shut down firewall b in azure everything works fine.  Any ideas?