SSL VPN Connection - 455 Permission Denied

Fortigate 80E with firmware v5.6 running.

There is a user group created called VPNUsers that is an LDAP lookup to AD on an internal server The VPN Users group is assigned to the SSL Portal called tunnel-access.

If users are in the appropriate group in AD, they can connect without any issue.

I need to create a "restricted" user for a contractor, so I created a user called contractor locally on the Fortigate

I create a user group called RestrictedUsers and added contractor I add RestrictedUsers to that same portal assignment that VPNUsers is assigned to I try to login and get -455 Permission Denied

If I start over from scratch, create a local firewall user called contractor, edit the VPNUsers group, and add contractor to that group, the contractor user can successfully log in.

If I remove contractor from the VPNUsers group, then add it directly to the tunnel-access portal assignment... 455 permission denied.

I am pulling my hair out trying to figure out why it is doing what it is doing.

Any suggestions would be very much appreciated!