is it possible that the user group is used in some admin configuration on some FortiGate to allow admins to log in with their LDAP credentials?
This would not show as reference, as admin settings are Device Manager config, not Policy&Objects config, but could still prevent deletion, as there are implicit dependencies in place.
In addition, if for some reason you can't find references for the LDAP server configuration in FortiManager, you could check the FortiGate(s) this LDAP server was pushed to and determine if there is any config outside the purview of FortiManager that still uses the configuration and might be preventing deletion.
If you are still unsure, you can raise a ticket with the FortiManager team to get further assistance :).
+++ Divide by Cucumber Error. Please Reinstall Universe and Reboot +++