FortiGate RADIUS Server Connectivity issue

[chethan]

Hello Everyone,

 From past couple of days we are facing this issue with FortiAuthenticator (RADIUS Server) Connection Status issue.

The connection status keeps on loading.

[ul]

The server secret is correct.

Can ping FortiAuthenticator from FortiGate.

Single factor authentication is working for remote RADIUS users.

FortiToken 2FA for remote RADIUS users is not working.

It was working before 4 days, no configuration was changed in between.[/ul]

*Image Attached for reference* 

Issue Resolving Urgency is high.

Thank you.

[xsilver_FTNT]

Hi,

"Issue Resolving Urgency is high." :)  So you have a support contract and did open TAC ticket, and got that resolved already, right?

Otherwise you haven't figured out that Forum is basically users-to-users support and we are helping here on voluntary basis.

What would definitely help you and us to assess situation is:

---

- software versions on both ends, FortiGate and FortiAuthenticator

- FortiOS debug output 'diag debug application fnbamd 7' .. as that's the daemon responsible for almost all outer auth then debug might suggest

- another hint might come from packet capture, like .. diag sniff pack any 'host <your-secret-FAC-IP> and port not 8000' 4 0 a

To see at least if there is any request and response, or better with verbosity 6 instead of 4, or use GUI Packet capture, ideally on both ends to confirm traffic is flowing unchanged so there is no middle man/firewall there.

Ideally there is supposed to be test RADIUS Access-Request from FGT to FAC with User-Name=test01 , followed by RADIUS response, probably Access-Reject .. to simply check there IS RADIUS service running on FAC

- if single factor (password based probably) auth via FAC is working and 2FA is not, then it might be separate issue and I would suggest to have a look to respective RADIUS Service Policy on FAC.

-- Is your requests matching right/intended policy (as order of the policies does matter)?

-- Isn't there set "Password-only authentication" in side of 'Authentication factors'?

-- Does the test user even have any 2FA set?

- how is the authentication "for remote RADIUS users." set on FGT, all the requests goes to FAC, or are there locally defined users on FGT with 'set type radius' and pointing to FAC, if so then is the used username matching the record on FGT, because by default is FGT case-sensitive (as any Unix system). And that could be overriden in user config via 'set username-case-sensitivity'. As that's one of common caveats where non matching username + remote server in the same user-group on FGT (or worst: radius server on FGT with 'set all-usergruop enable') makes unintended fallback to remote server and avoid 2FA this way. But that's configuration mistake.

- on FAC check that FGT is properly set as RADIUS Service Client and used in some Policy, as FAC will NOT respond to anyone without being set as known Client first!

- and lately, check on FAC side .. GUI logs, and in GUI debug https://<FAC-IP>/debug/radius/  even basic output there might provide some hints and you can even turn RADIUS server to debug mode for increased verbosity of the log (I would not keep it in debug mode for normal operations so turn it off after troubleshooting).

[chethan]

Thank you for replying,

"I thought to find an answer on the forum before raising the support ticket".

- The users are remote LDAP users (users are not local to FortiGate).

- 2FA is set using FortiToken Mobile on FortiAuthenticator for users and it is how they are authenticated from its initial deployment.

- Single factor works.

- We are observing this from past 4 - 5 days, even though no configuration was changed in between (2FA was working fine before this).

- On FortiAuthenticator Logs, I can see it is authenticating the Token as well successfully.

I have run the required diagnostic commands on FortiGate:

After entering the token, I can see that the traffic goes from FortiGate to FortiAuthenticator but never returns. On FortiGate it waits for the response from FortiAuthenticator for long enough to fail from timeout.

I have attached the image below, It says "can't contact RADIUS server" even thought single factor still works.

[ul]

We have checked the server secret on both FortiAuthenticator and FortiGate it is same.

On FortiAuthenticator policy, It is set to "Mandatory two-factor authentication". 

There is only one radius client (FortiGate) and it is used in the policy (only one policy defined).

No local users are created on FortiGate. All the requests go to FAC.[/ul]

Thank you.

[xsilver_FTNT]

Excuse me but that's pretty weird.

IF you have FGT pointing to FAC, where there is a single RADIUS Service / Policy .. allowing FGT as the only client, pointing users to realm which is LDAP based (probably to AD), AND the policy has "Mandatory two-factor authentication".

Then how could it be that "Single factor works" as you state ?!

If mandatory 2FA is set, and by any chance you will use user who has no 2FA set and fully activated on FAC, then such logon attempt will not ask for second factor, but it will also fail. Exactly because there was not possible to check second factor. Regardless of fact that same login user/pass will work if "Verify all configured authentication factors" would be set, so user without token will pass and user with token will be forced to use it (and fail without it).

So, I would suggest to either open technical ticket, or follow your config ... how is RADIUS defined on FGT, how it's used in policy on FGT, check FNBAMD debug to see that right server is contacted (FAC), thne continue on FAC, in policy, all the sections, how backend is defined, what is default realm. If there is trully just one policy for that FGT as RADIUS Client, and what's on /debug/radius. Simply use packet capture or CLI 'exec tcpdump..' to confirm that FAC seen Access-Request, and sent back Challenge-Request for token .. and on FGT that such challenge was received and responded with new Access-Request where now User-Password APV contain token code.

Alternatively you can trigger such user authentication from simple SSLVPN or even directly from CLI on FGT via 'diag test authserver radius <RADIUS-SERVER-NAME-from-ConfigUserRadius> pap <test-user-name> <password>'.

If that test user is equipped with token then you should get token request even on FGTs' CLI.

For example: (user A with token, user B without token, but policy enforce 2FA use)

c-esx02 # diag test authserver radius C4 pap userb tester authenticate 'userb' against 'pap' failed, assigned_rad_session_id=264016458 session_timeout=0 secs idle_timeout=0 secs!

c-esx02 # diag test authserver radius C4 pap usera tester Token Code:****** authenticate 'usera' against 'pap' succeeded, server=primary assigned_rad_session_id=264016465 session_timeout=0 secs idle_timeout=0 secs!

c-esx02 #

[chethan]

Hello,

First factor works. By that I mean, when the users enters the username and password FAC accepts it and challenges the user for token. Now the user enters the token, on FAC logs we can see the message "Remote LDAP user authentication with FortiToken successful". But this response it not reaching FortiGate.

I have attached the .txt file (I have changed the IP address to a.b.c.d).

You can see that after user enters token, it waits for 300 sec (Timeout period) and gets timed out.

[emnoc]

So what I would do is grab a pcap of the radius traffic and inspect the content of the request accept messages. You can use radiussniff or radiusdump to decode the packet access-request  or wireshark.

I would do a pcap collection while testing fromt the fgt cli with 2 or 3 accounts 

Ken Felix

[xsilver_FTNT]

I tlooks like init phase is sort of working, and user data were sent, then token challenged .. 

"fnbamd_auth_handle_radius_result-->Result for radius svr 'FortiAuthenticator' a.b.c.d(1) is 2"

Then token acquired on CLI and sent up to FAC

2021-07-07 19:54:31 Token Code:******

2021-07-07 19:54:52 [2339] handle_req-Rcvd chal rsp for req 85445526

2021-07-07 19:54:52 [1310] __fnbamd_rad_send-Sent radius req to server 'FortiAuthenticator': fd=15, IP=a.b.c.d(a.b.c.d:1812) code=1 id=3 7 len=102 user="username" using PAP

Now, check FAC side /debug/radius/ for that user and request (or make a new test to see online).

"FAC logs we can see the message "Remote LDAP user authentication with FortiToken successful". But this response it not reaching FortiGate."

- no more details, just this one liner .. but it shows that backend LDAP was contacted and auth with username + pass succeeded

- however this message is not carried to FGT, as expected, because it's FAC internal business with LDAP, nothing FGT needs to know about. Only final auth result, Access-Accept or Access-Reject is carried back to FGT from FAC.

- based on that LDAP the token is challenged (unless you would have "PCI DSS 3.2 two-factor authentication" ON), so this works till now

- FGT seems to be sending token back to FAC

... so now you need to check if token is received on FAC, processed and what's the result code/reaction on FAC side

BTW: tokens are handled by FAC or is that chained auth and tokens are handled by some other 3rd party device like RSA tokens ?

Asking because if those are chained, then timeout/response-delay to that 3rd party 2FA server might be the cause.

If those are FortiToken of any sort on FAC, especially mobile tokens, then check token listing FAC/Authentication/UserManagement/FortiTokens .. find the token you are using, hover over Drift column and use "Sync" which should appear there, to sync time / adjust drift between token and FAC. That is crucial part of TOTP, but it should be visible from log that token is out of sync too badly that drift detection did not caught and auto-adjusted.

[chethan]

Hello, 

Thank you for your insightful inputs in this issue.

The tokens are handled by FAC itself and are not chained.

Yes, FortiAuth Logs show that it is authenticating the Tokens as well. This log "Remote LDAP user authentication with FortiToken successful" is after the user enters the token.

On FortiAuth: "Access-Accept (2), id: 0x03, Authenticator: 717a0a467199fb65138a74537bc1c1cd"

After verifying the token (Access-Accept), but this response is not reaching FortiGate. (FGT waits until it gets timed out). 

EDIT: We have raised a ticket with Fortinet, The Fortinet support engineer is looking into the issue now.

[xsilver_FTNT]

So, if FAC verified credentials and token and issued Access-Accept, which was sent form FAC (seen in packet capture on wire), but never received on FGT (never seen in packet capture on wire).

Then it is obviously NOT a problem in authentication but with missing packets .. so, jitter, packet loss tests, check intermediate firewalls, or DOS on FGT .. simply track that lost packet (the one with Access-Accept).

Isn't there any weirdo setup like asymmetric routing ?

[emnoc]

Yeah that last part make me believe your routing is dorked. On the Fortiauthenticator do we have any  means to test a login from the authenticator cli ? You might can test the a user and token from the cli to check that MFA is working. If it works and your ack than the radius client to server ( fgt to fac ) connectivity issues might be the issue for OP problems.

Just and ideal

Ken Felix