ADGrp .. "config user adgrp" entries are either Group Filter records for standalone Collector Agent pushed as per-FortiGates' serial number specific filter (when your FGT do have LDAP in 'config user fsso' for respective Collector), or pulled from Collector Agent, if that collector do have per-FGT specific filter, or Default filter set and FGT do NOT have LDAP in settings.
In either case how the records got to 'config user adgrp' those are USER GROUP records !!
Therefore, those should NOT contain specific users or devices, those should point to GROUP type of objects which sort of consolidate all possible candidates.
Because FSSO is based on group membership.
Collector can read group membership from AD.
Collector do not need to filter every single user via group filter and FGT then do not need to consolidate those single adgrp records into 'config user group' fsso type!
It is not intended to duplicate groups known/defined on AD and I would consider this as configuration error.
Goal of FSSO Group filter is to learn group membership from AD and let AD Admins to "drive" from AD level who is eligible to access what and through firewall (FortiGate), via group membership processed by Collector and users' membership shared to FGT (which then drive access privileges based on groups).
So, if you want to grant access to specific users, then group them to some specific AD group.
Then add this group to Group Filter on Collector (to push to FGT), or add this to FGT where FSSO Connector do have LDAP in config (which is there solely for this purpose, as FGT do not use it for group verification but just config).
Then use above gained adgrp record in firewall user group type fsso, and this can be then used in policies (both on FGT).
That's the way how to use Group filter.
Tom xSilver, planet Earth, over and out!