FortiGate Next Generation Firewall utilizes purpose-built security processors and threat intelligence security services from FortiGuard labs to deliver top-rated protection and high performance, including encrypted traffic.
Article Id 194213

This article explains a limitation inherent to IKE when trying to configure multiple IPSec phase1 in Pre-Shared Key (PSK) mode.

If you configure more than one *dialup* phase1 in Main Mode PSK on the same FortiGate port (IP),  then the same phase1 will always be matched by ALL dialers. The error will show up in the third Main Mode round (authentication) as per the example below :

2009-11-04 16:00:00 device_id=FGTxxx  log_id=0101023003 type=event subtype=ipsec pri=error loc_port=500 rem_port=500 out_if="wan1" vpn_tunnel="branchOffice" cookies=asd2345sdf4sdf345 action=negotiate status=negotiate_error msg="Negotiate SA Error: probable pre-shared secret mismatch"

When multiple *dialup* Main Mode PSK phase1 exist, IKE has no way to know which phase1 must be matched upon receipt of the first main mode packet :
- the source IP (dialer's IP) cannot be used as a distinguisher (any source IP is allowed for dialup phase1)
- the destination IP (FGT's IP) cannot be used as a distinguisher (we have assumed that all dialup phase1 were bound to the same FGT IP)
- there isn't any payload in the first Main Mode packet which would allow to select a specific phase1

Consequently, the first phase1 in IKE internal list is always matched.
This is not a FortiOS limitation

The PSK itself is part of the phase1 root key (SKEYID) calculation.
This root key is then derived into two keys (SKEYID_e, SKEYID_a) which are used for the phase1 encryption and authentication.
Phase1 keys are therefore dependent on the PSK.

When the phase1 initiator sends the MI3 message (fifth Main Mode message) it is protected with its locally calculated SKEYID_e/SKEYID_a. If the phase1 which was chosen by the responder is not the correct one, then SKEYID_e and SKEYID_a on the responder are different from those on the initiator therefore leading to a failure ('probable PSK mismatch' in the 'debug ike') because the responder is unable to decipher the initiator's packet.

All FortiOS

To work around this Main Mode PSK limitation for multiple dialup connections :

  • use aggressive mode with ID (ID is sent in clear text in the first Aggressive Mode message and can therefore be used by the responder to select the correct phase1)
  • use Main Mode with RSASIG

Main Mode RSASIG doesn't suffer this limitation and still offers ID protection :
Upon receipt of the first
Main Mode message, the IKE responder picks up the first dialup Main Mode RSASIG phase1 in its list. If it is not the correct phase1, it is not an issue.
When MI3 message will reach the IKE responder, it will be able to authenticate and to decipher it because SKEYID_a/e keys don't depend on the phase1 specification (the root SKEYID is based on payloads that were exchanged between each peer (nonces) and on the Diffie-Hellman private shared key that was locally computed by each peer).
Once the MI3 message is decrypted, the IKE responder finally knows the ID of the initiator (which by default is the DN of the initiator certificate) and can therefore "fallback" to the correct phase1 if needed.