IKEv2 digital signature with rsa-pss-sha2_256 fails

[strongX509]

I'm enabling IKEv2 digital signatures with rsa-pss on a FortiGate VPN Gateway:

config vpn ipsec phase1-interface

    edit "xyz"

        ...

       set digital-signature-auth enable
       set signature-hash-alg sha1 sha2-256 sha2-384
       set rsa-signature-format pss

    next

end

When the peer sends an client certificate and an IKEv2 AUTH payload with an RSA-PSS SHA1 signature then everything works well:

ike 0:dbmas:187: certificate validation succeeded
ike 0:dbmas:187: signature verification succeeded using rsassa-pss-with-empty-params, oid=1.2.840.113549.1.1.10
ike 0:dbmas:187: auth verify done
ike 0:dbmas:187: responder AUTH continuation
ike 0:dbmas:187: authentication succeeded

When the IKEv2 AUTH payload contains an RSA-PSS SHA2-256 signature with a default salt size of 32 bytes then the following error occurs:

ike 0:dbmas:190: certificate validation succeeded
ike 0:dbmas:190: signature verification failed

A client certificate with an RSA-PSS SHA2-256 signature  seems to be accepted, though, but the IKEv2 AUTH signature verification still fails:

ike 0:dbmas:193: certificate validation succeeded
ike 0:dbmas:193: signature verification failed

The FortiGate VPN gateway runs FortiOS v7.0.1 and the VPN client runs strongSwan 5.9.7. 

[strongX509]

Ok, I found the reason for the RSA-PSS-SHA2_256 signature verification error. The FortiOS implements the default ASN.1 AlgorithmIdentifier defined in Appendix A.4.3 of RFC 7427:

   0000 : SEQUENCE
   0002 :   OBJECT IDENTIFIER  RSASSA-PSS (1.2.840.113549.1.1.10)
   000d :   SEQUENCE
   000f :     CONTEXT 0
   0011 :       SEQUENCE
   0013 :         OBJECT IDENTIFIER  id-sha256 (2.16.840.1.101.3.4.2.1)
   001e :         NULL
   0020 :     CONTEXT 1
   0022 :       SEQUENCE
   0024 :         OBJECT IDENTIFIER  1.2.840.113549.1.1.8
   002f :         SEQUENCE
   0031 :           OBJECT IDENTIFIER id-sha256 (2.16.840.1.101.3.4.2.1)
   003c :           NULL
   003e :     CONTEXT 2
   0040 :       INTEGER   0x20 (6 bits)
   0043 :     CONTEXT 3
   0045 :       INTEGER   0x1 (1 bits)

   Name = RSASSA-PSS with sha-256, oid = 1.2.840.113549.1.1.10
   Length = 72
   0000: 3046 0609 2a86 4886 f70d 0101 0a30 39a0
   0010: 0f30 0d06 0960 8648 0165 0304 0201 0500
   0020: a11c 301a 0609 2a86 4886 f70d 0101 0830
   0030: 0d06 0960 8648 0165 0304 0201 0500 a203
   0040: 0201 20a3 0302 0101

  Section 3.1 of RFC 4055 gives the complete ASN.1 definition as 

     RSASSA-PSS-params  ::=  SEQUENCE  {
         hashAlgorithm      [0] HashAlgorithm DEFAULT
                                   sha1Identifier,
         maskGenAlgorithm   [1] MaskGenAlgorithm DEFAULT
                                   mgf1SHA1Identifier,
         saltLength         [2] INTEGER DEFAULT 20,
         trailerField       [3] INTEGER DEFAULT 1  }

All the CONTEXT fields have DEFAULT values. Our strongSwan implementation omits the trailerField, assuming a default value of 1. The notes in the Errata of RFC  7427 say explicitly:

Implementations that perform signature validation MUST recognize both a present trailerField field with value 1 and an absent trailerField field. 

Thus it would be very helpful if FortiOS would recognize the second valid variant of

0000: 3041 0609 2A86 4886 F70D 0101 0A30 34A0
0010: 0F30 0D06 0960 8648 0165 0304 0201 0500
0020: A11C 301A 0609 2A86 4886 F70D 0101 0830
0030: 0D06 0960 8648 0165 0304 0201 0500 A203
0040: 0201 20

 with a length of 67 bytes. The same applies accordingly to the RSA-PSS-SHA2_384 and RSA-PSS_SHA2_512 encodings.

[strongX509 · Accepted solution]

As a workaround until Fortinet adds the alternative RSA-PSS algorithmIdentifiers for SHA256, SHA384, and SHA512, the strongswan.conf option

charon.rsa_pss_trailerfield = yes

will generate an algorithmIdentifier containing an explicit trailerfield entry. This feature will be available with strongSwan 5.9.8.