Controlling traffic to networks over an IPSEC VPN using users and rules

Hi All

We have setup a basic IPSEC VPN using Forticlient and the Wizards in a Fortigate 100D.

Rules allow traffic to two different subnets internally from the VPN connected users. 

However, we'd like to only allow certain users to access one subnet. As soon as I add any user at all to the firewall rule I lose all ability to talk to that network for all users, even though they are authenticated.

Am I missing something?