FortiGate Next Generation Firewall utilizes purpose-built security processors and threat intelligence security services from FortiGuard labs to deliver top-rated protection and high performance, including encrypted traffic.
Article Id 197897
This article describes how to troubleshoot RSSO issue.

With RSSO, a FortiGate can authenticate users who have authenticated on a remote RADIUS server.
Based on which user group the user belongs to, the security policy
applies the appropriate UTM profiles.

Below step for troubleshoot RSSO issue in the FortiGate.

1) Run the below debug in FortiGate to check radius accounting packet.
# diagnose  debug  application  fnbamd -1
# diagnose  debug  application  radiusd -1
# diagnose  debug  en
Sample Output :
Received radius accounting eventvd 0:root Add/Update auth logon for IP for user sumit DB 0 insert [ep='sumit' pg='group1'
ip=''] success

> RSSO group name is pg='group1', ep='sumit' is user-name.
2) Command to verify RSSO user auth list in FortiGate.
# diagnose firewall auth list, sumit type: rsso, id: 0, duration: 13, idled: 13
        flag(10): radius  server: root
        packets: in 0 out 0, bytes: in 0 out 0
        group_id: 3
        group_name: RS                         <----- Tagged group.
----- 1 listed, 0 filtered ------
3) Modify the default value for rsso-endpoint-attribute might work in common remote access scenarios where users are identified by their unique Calling-Station-Id,
 but in other scenarios the user name might be in a different attribute.
# config user radius
    edit "RSSO Agent"
        set rsso-endpoint-attribute <attribute>
Example, In below case users information is passing in 'User-Name' attribute.
Modify the below rsso-endpoint-attribute from the FortiGate.

# config user radius
edit "RSSO Agent"
       set rsso-endpoint-attribute User-Name    <---- change attribute.