Technical Tip: FortiGate RADIUS authentication not working with Microsoft NPS when using accented or UTF-8 special characters in passwords

Scope

FortiGate.

Solution

The setup is as follows: 

1. FortiGate is configured as a RADIUS Client.
2. Microsoft NPS is configured as a RADIUS server.
3. The following details are being used to log in: 

User: test.

Password: ä12345.

When trying to use accented characters as above in the password, the authentication fails. 

diagnose debug console timestamp enable

diagnose debug application fnbamd -1

diagnose debug enable

diagnose test authserver radius Rad_server ms-chapv2 test ä12345

[1329] __fnbamd_rad_send-Sent radius req to server 'Rad_server': fd=14, IP=10.10.1.193(10.10.1.193:1812) code=1 id=17 len=181 user="test" using MS-CHAPv2
[320] radius_server_auth-Timer of rad 'Rad_server' is added
[596] create_auth_session-Total 1 server(s) to try
[1388] fnbamd_auth_handle_radius_result-Timer of rad 'Rad_server' is deleted
[1797] fnbamd_radius_auth_validate_pkt-RADIUS resp code 3
[416] extract_chap_error-CHAP err: E=691 R=0 V=3
[1414] fnbamd_auth_handle_radius_result --> Result for RADIUS svr 'Rad_server' 10.10.1.193(1) is 1.
[217] fnbamd_comm_send_result-Sending result 1 (nid 0) for req 1602112804, len=3084

To disable the current debug:

diagnose debug disable

diagnose debug reset

This is a limitation for the Microsoft NPS, where accented characters, if used in passwords, will lead to this limitation.  The same occurs with symbols like the euro €.

This is explained in Network Policy Server Best Practices.

Microsoft NPS is not encoding RADIUS passwords in UTF-8. It is instead encoding it in Extended ASCII. Since it is not complying with RFC2865, the FortiGate authentication daemon would receive a password that would be a mismatch, and authentication would fail. 

The following options are currently available for password encoding in FortiGate :

config user radius

    edit <radius name>

        set password-encoding ?

        auto          Use original password encoding.

        ISO-8859-1    Use ISO-8859-1 password encoding.

Note:

If using Microsoft NPS with FortiGate as a RADIUS server, it is advised not to use any accented characters or special symbols in the password. Strictly use Symbols and punctuation of ISO-8859-1.

Related articles:

• Troubleshooting Tip: Possible reason for RADIUS Reject code 3
• Technical Tip: RADIUS error codes