[SOLVED] Packets for port 5038 go through S2S tunnel but don't get routed localy

Model number: 100D (x2 in HA cluster)

Firmware: v5.2.9,build736

Issue:

We have a site to site IPSec tunnel to our customer created using the the wizard and a "Site to Site - Fortigate" template (the other end is a Fortigate 90d running v5.4.1,build1064).

The customer site has two networks:

10.14.48.0/24 - for computers

10.14.50.0/24 - for VOIP phones

Our end:

vlan internal network (number 12) with a PBX server

There are rules in place for VOIP traffic and the SIP helper is disabled on the 100D (system wide) - VOIP works fine over the tunnel to the PBX server on vlan12. Now I have configured rules on both ends so that the computer network is able to talk to the PBX server on vlan12 using the TCP port 5038.

My issue is that the traffic exits the customer Fortigate (comes in on the internal network and goes out the tunnel interface) and arrives at our Fortigate on the tunnel interface, however it does NOT get forwarded to the vlan12 interface where the PBX server is located. Again, the VOIP traffic from the phone network DOES get forwarded to the vlan12 interface.

I have attached a screenshot from the UI  showing the problematic policy I have and an example of the exact same working policy with different ports .

Here (since I can't attach more than one image) are all the screenshots with an example of a working policy and the problematic policy, where the difference is only the ports (services) There are also screenshots of packet capture from the customer Fortigate and our Fortigate showing the traffic flow.

Any help would be appreciated.

~levi