one out of 7 IPsec VPN's unstable

[lamalord]

I have the following situation:

In my datacenter, i have a PFsense set up that is connected to 7 fortigate firewalls (30D - v5.6.8 build 1672 (GA)). 

My issue is the following: 

one of them is very unstable (it will rarely connect, and when it does, it works but goes disconnected again for 30 minutes). 

the unstable one however is connected to another fortigate that doesn't have the problem. 

My set up is the following: 

Fortigate 30D (wan ip 192.168.1.33) <-> (Full port forward) ISP Modem  ----- INTERNET ----- MAIN FIREWALL -- PFSense 

NAT T is forced between the 2 sites. 

my pfsense is trying to send packets to my fortigate: 

sending packet: from PUBLIC_WAN_IP_PFSENSE[500] to PUBLIC_WAN_IP_FORTIGATE[500] (360 bytes)

The only strange thing i noticed on my pfsense is that it sends packets on port 500 (and not 4500 like all of the others). 

My fortigate keeps sending IKE keep alives and NAT keep-alive 5 (from its private wan IP to my public wan of the pfsense) without success.

When i try to bring it up on my fortigate (phase 1), i see that the latest message is 'SENT IKE MSG (p2_RETRANSMIT): (from its private wan IP:4500 to my public wan ip of pfsense:4500)

Could this port mismatch be the cause of my instable VPN?