incoming IPSec VPN traffic only works with any source interface

This is driving me mad.

I have set up an IPSec VPN and want to limit it to a certain set of destinations.

As I have two WAN links up, I'm connected on one and playing with the VPN settings of the other.

I thought I understood how this works, but I'm now utterly baffled.

I'm connecting with the FortiClient. I have my static routes pushed through OK so I can route to the destination network I want.

Now when it comes to policies, I have set up a policy which has from the remote tunnel to the destination but this doesn't work.

If I change the policy so the source interface is "any", it works.

If I then change the policy so the source interface is the Remote Access WAN interface that's set up, it doesn't work, traffic gets dropped and is picked up by the default deny policy at the bottom.

When I have the policy configured so that the source interface is "any" and it works, if I look at the policy logs, I can see the source interface is my RA WAN.

So why doesn't it work when I set it to that interface? Furthermore, if I manually select ALL the interfaces that are up, it still doesn't work! It only lets traffic through when the source is "any".

I'm completely stumped!