RADIUS test connectivity fails possibly due faulty Message-Authenticator Java implementation

Hi

We hope this board has some java developer as well:

We developed a Java based RADIUS server but since firmware 7.2.10 customers experience issue.

We know, we have to include the Message-Authenticator Attribute (80) in the response package. To generate the Message-Authenticator, we use the following Java-code:

public static byte[] generateMessageAuthenticator3(byte[] sharedSecret, int packetCode, int packetIdentifier, int packetLength, byte[] requestAuthenticator, byte[] attributes) {  try {    Mac mac = Mac.getInstance("HmacMD5");    mac.init(new SecretKeySpec(sharedSecret, "HmacMD5"));     mac.update((byte) packetCode);    mac.update((byte) packetIdentifier);    mac.update((byte) (packetLength >> 8));    mac.update((byte) (packetLength & 0x0ff));    mac.update(requestAuthenticator, 0, requestAuthenticator.length);    mac.update(attributes, 0, attributes.length);    return mac.doFinal(); } catch (NoSuchAlgorithmException ex) {    ex.printStackTrace();    return null; } catch (InvalidKeyException ex) {    ex.printStackTrace();    return null; } }

We know, the shareSecret is correct on both ends, because we can decrypt the password if we hit "Test User Credentials". PacketType and PacketIdentifier are as well, obvious. The PacketLength is the length of the RadiusPacket, the sum of the length of each RadiusAttribut + 1 (Code) + 1 (Identifier) + 2 (RP-Length) + 16 (RP-Authenticator). The RequestAuthenticator is the same byte-stream the FortiGate sends with its Access-Request.

let's see the byte-stream the FortiGate sends:

[1, 0, 0, 64, -20, 25, 37, -38, -58, 89, 122, -48, -76, 26, -49, -76, -65, -15, -59, -122, 32, 18, 70, 71, 86, 77, 69, 86, 75, 89, 71, 65, 79, 81, 67, 73, 66, 49, 1, 8, 116, 101, 115, 116, 48, 49, 2, 18, -53, 0, 102, 34, -62, 74, 124, -127, 40, 100, 56, 53, -107, 36, -1, -55]

red: 1=Code, 0=Identifier, 64 = packet length

blue = 16 bytes of Request-Authenticator

green = AttributeID 32 =  NAS-Identifier

purple = AttributeID 1 = Username

black = AttributeID 2 = Password

For the response RadiusPacket for this request, we use the following data stream as a "template":

[2, 0, 0, 76, -107, -92, -73, -115, -60, 117, 7, 112, 108, 16, -20, -20, -69, 40, 101, -102, 18, 38, 65, 117, 116, 104, 101, 110, 116, 105, 99, 97, 116, 105, 111, 110, 32, 83, 101, 114, 118, 101, 114, 32, 110, 111, 116, 32, 97, 118, 97, 105, 108, 97, 98, 108, 101, 33, 80, 18, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

red: 2 = Access-Accept, 0 = Identifier, 76 packet length

orange: 16 bytes of Response-Authenticator, generated according RFC2865

green: AttributeID 18 = Reply-Message, 38 bytes

purple: AttributeID 80 = Message-Authenticator, 18 bytes (zeroed)

now we apply the generateMessageAuthenticator()-methods declared above on our response RadiusPacket:

sharedSecret = MySecret.getBytes();

PacketType = 2 (Access-Accept), see red of Response-RP

PacketIdentifier = 0, see red of Response-RP

PacketLength = 76, see red of Response-RP

RequestAuthenticator = [-20, 25, 37, -38, -58, 89, 122, -48, -76, 26, -49, -76, -65, -15, -59, -122], blue, see of Request-RP

Attributes = [18, 38, 65, 117, 116, 104, 101, 110, 116, 105, 99, 97, 116, 105, 111, 110, 32, 83, 101, 114, 118, 101, 114, 32, 110, 111, 116, 32, 97, 118, 97, 105, 108, 97, 98, 108, 101, 33, 80, 18, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], see Response-RP

This results in a Message-Authenticator-ByteStream: [-123, 104, 63, 100, -95, -125, 109, 3, -81, -37, -108, 121, -36, 47, -34, 4]

We replace this Message-Authenticator-ByteStream into the initial Reponse-RP, where the zero-placeholder were:

[2, 0, 0, 76, -107, -92, -73, -115, -60, 117, 7, 112, 108, 16, -20, -20, -69, 40, 101, -102, 18, 38, 65, 117, 116, 104, 101, 110, 116, 105, 99, 97, 116, 105, 111, 110, 32, 83, 101, 114, 118, 101, 114, 32, 110, 111, 116, 32, 97, 118, 97, 105, 108, 97, 98, 108, 101, 33, 80, 18, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

[2, 0, 0, 76, -107, -92, -73, -115, -60, 117, 7, 112, 108, 16, -20, -20, -69, 40, 101, -102, 18, 38, 65, 117, 116, 104, 101, 110, 116, 105, 99, 97, 116, 105, 111, 110, 32, 83, 101, 114, 118, 101, 114, 32, 110, 111, 116, 32, 97, 118, 97, 105, 108, 97, 98, 108, 101, 33, 80, 18, -123, 104, 63, 100, -95, -125, 109, 3, -81, -37, -108, 121, -36, 47, -34, 4]

This is our Response-RP. But the Fortigate-WebGUI has always the error: Invalid secret for the server  Error in the CLI log:

[342] fnbamd_create_radius_socket-Opened radius socket 13

[2894] receive_parse_radius_check_response-The shared secret is incorrect.

But as stated above, we know, the shared secret is correct, as we can decode the Password!

We have seen other logs, where the Message-Authenticator is missing:

maybe something is off with our Message-Authenticator, but we cannot find the issue.

So we hope, some of you stumbled upon this issue and know exactly what we are doing wrong!

thank you!

Sven