Hi Benson,
I don't think it is possible, if I get you right - you limit access to the VPN SSL portal(s) by source IP address of the clients? If so, then this configuration is global for the SSL VPN service, and I don't see an option to make it otherwise. Only separate VDOMs for each group of users.
Realms allow separation per User Group/URL, but AFTER ANY client from the Internet reached the VPN SSL port already. Basically it is the same as mapping different User Groups on Fortigate to different portals with no limit to specific hosts - if you limit (or not) access to Specific Hosts, you limit (or not, accordingly) access to all portals/realms at once. This is configuration-wise, I haven't tried to validate on actual Fortigate though.
HTH,
Yuri