FortiGate Next Generation Firewall utilizes purpose-built security processors and threat intelligence security services from FortiGuard labs to deliver top-rated protection and high performance, including encrypted traffic.
Anti-replay from IPsec phase2 is a local setting. It is not negotiated between IPsec peers therefore it does not impact the establishment of the tunnel.
However, anti-replay could affect the traffic through the tunnel, and could be the cause of ESP packets being dropped. It is recommended to have the same anti-replay setting on both the local and peer IPsec.
Anti-replay mechanism is based on sequence number to mark the ESP packets. The sequence number is in clear-text so it should only be trusted if authentication is enabled. If anti-replay is enabled for the inbound IPsec SA or phase2, the sequence number is checked by the recipient.
The sender increases the sequence number for each sending ESP packets by one. The receiver compares the received sequence number and adjusts the sliding anti-replay window. If any encrypted packet arrives out of order and not in the window range, the FortiGate unit discards it. The receiver can reject old or duplicate packets to protect itself against replay attacks, against an unauthorized party to intercept and replay a series of IPsec packets.
As a receiver, FortiGate will log the discarded packets with the following Event Log: "Invalid ESP packet detected (replayed packet)" message.
As the anti-replay is not negotiated, FortiGate will act according to its local setting of anti-replay.
In cluster mode FortiGate HA, in case of a failover, depending on the FortiOS versions, NP interfaces and settings, the sequence numbers are not synchronized between the master and the slave units and could be the cause of ESP packets being dropped by the peer.
If anti-replay is disabled on the local IPsec unit but enabled on the peer, then the sequence number from local FortiGate should not enter in the replay windows of the IPsec peer, which will discard it. In this situation, the IPsec tunnels are up on both IPsec units. The Tx ESP packet counter is increasing for this phase2 but there are probably no new Rx packets.
In FortiOS V5.2, the ESP sequence numbers are NOT synchronized between master HA and slave. When anti-replay is disabled, after the HA failover, the new master will start sending packets with a sequence number of 1. While if anti-replay is enabled, the FortiGate will force a rekey and the IPsec negotiation.
Starting from FortiOS V5.4, the ESP sequence numbers are synchronized between master HA and slave depending on the parameter from the Phase1 "set ha-sync-esp-seqno enable | disable" (enabled by default)
As from FortiOS V5.2, if IPsec is configured with IKEv2, and according to the support and negotiation of RFC 6311, the local and peer FortiGates will exchange the message-ID field to protect again replay.