Technical Tip: IPsec VPN traffic dropped as 'anti-spoof check failed, drop' after the upgrade to v7.4.2+
Description
This article describes the new anti-spoof IPsec VPN logic that was introduced in FortiOS v7.4.2. If proper conditions are not met, the traffic may be dropped as 'anti-spoof check failed, drop'.
Scope
FortiGate v7.4.2+.
Solution
When a VPN device (such as the FortiGate) transmits traffic to a remote peer via IPsec, that traffic must a) match a valid traffic selector that both VPN peers agreed upon as part of Phase2 negotiation, and b) must be transmitted using the correct Security Parameter Index (SPI) value and encrypt/decrypt keys for that traffic selector.
In FortiOS v7.4.1 and earlier, the FortiGate did check the first condition (traffic must match a valid, established set of traffic selectors), but it did not check the second condition. This meant that remote VPN peers could technically send traffic to the FortiGate using the wrong traffic selector as long as they both shared at least one valid selector that existed for that tunnel that would allow the traffic.
However, starting from FortiOS v7.4.2, the FortiGate now enforces anti-spoof checks for the second condition. This means that when remote peers send traffic for an allowed Source to an allowed Destination but using the wrong traffic selectors (which means the wrong encrypt/decrypt keys and the wrong SPI in the ESP packet), then the FortiGate will drop that traffic.
As a quick example:
- FortiGate and Remote Peer have two established traffic selectors (Selector_A and Selector_B) configured under the IPsec Phase2 settings.
- Remote Peer forwards traffic over the VPN that is allowed by Selector_A, but it uses Selector_B's SPI in the outgoing ESP packet, as well as Selector_B's negotiated encryption keys to secure the traffic.
- FortiGate receives the ESP packet, sees that the SPI matches Selector_B, and decrypts the packet, where it sees that the packet's source IP and destination IP address are allowed by Selector_A.
- In v.7.4.1 and earlier, the FortiGate would simply accept this packet and forward it through.
- In v7.4.2 and later, the FortiGate now flags the mismatch and drops the packet, citing 'anti-spoof check failed, drop' in the debug flow output.
The following are more in-depth examples detailing how these two conditional checks operate, including example packet captures and accompanying debug flow output:
Case 1: Initiating packet from remote site arrives on the FortiGate and is dropped.
In this example, a client (10.10.50.2) at a remote site has sent an ICMP ping towards 10.20.50.2, which is behind the FortiGate. In the packet capture, the ICMP echo request is observed coming inbound on the TESTVPN tunnel, but is not forwarded through:
diagnose sniffer packet any "host 10.20.50.2 and icmp" 4 0 l
interfaces=[any]
filters=[ host 10.26.10.41 and icmp ]
2024-02-27 17:26:13.425459 TESTVPN in 10.10.50.2 -> 10.20.50.2: icmp: echo request
2024-02-27 17:26:14.443703 TESTVPN in 10.10.50.2 -> 10.20.50.2: icmp: echo request
2024-02-27 17:26:15.425146 TESTVPN in 10.10.50.2 -> 10.20.50.2: icmp: echo request
2024-02-27 17:26:16.435445 TESTVPN in 10.10.50.2 -> 10.20.50.2: icmp: echo request
2024-02-27 17:26:17.452851 TESTVPN in 10.10.50.2 -> 10.20.50.2: icmp: echo request
At the same time, the debug flow on the FortiGate shows the following output as packets arrive:
diagnose debug flow filter addr 10.20.50.2
diagnose debug flow filter proto 1
diagnose debug enable
diagnose debug flow trace start 100
2024-02-27 17:45:40 id=65308 trace_id=1311 func=print_pkt_detail line=5888 msg="vd-root:0 received a packet(proto=1, 10.10.50.2:53450->10.20.50.2:2048) tun_id=123.123.123.123 from TESTVPN. type=8, code=0, id=53450, seq=0."
2024-02-27 17:45:40 id=65308 trace_id=1311 func=ipsec_input4 line=279 msg="anti-spoof check failed, drop"
Checking the output of diagnose vpn tunnel list shows that the traffic selectors that have been agreed upon for the TEST VPN tunnel:
proxyid_num=2 child_num=0 refcnt=5 ilast=0 olast=3 ad=/0
stat: rxp=80022 txp=395 rxb=14809443 txb=25814
dpd: mode=on-idle on=1 idle=20000ms retry=3 count=0 seqno=36
natt: mode=none draft=0 interval=0 remote_port=0
fec: egress=0 ingress=0
proxyid=TESTVPN proto=0 sa=1 ref=3 serial=13
src: 0:0.0.0.0-255.255.255.255:0
dst: 0:192.168.10.0-192.168.10.3:0
SA: ref=3 options=10025 type=00 soft=0 mtu=1446 expire=1733/0B replaywin=0
seqno=8 esn=0 replaywin_lastseq=00000000 qat=0 rekey=0 hash_search_len=1
life: type=01 bytes=0/0 timeout=1767/1800
dec: spi=36123e0e esp=3des key=24 d569b167770be97121e5b0a6545519b52c991f526864ec12
ah=sha1 key=20 f028a15a42b0dcdd26ffc77838b4851928c875ea
enc: spi=dd113712 esp=3des key=24 991b59099c3f8dd69576fa102e3c308997b544a62a3fc2b8
ah=sha1 key=20 3494b9d54e11e14486554b0f0ad70522586b065a
dec:pkts/bytes=1365/263845, enc:pkts/bytes=7/824
npu_flag=00 npu_rgwy=123.123.123.123 npu_lgwy=123.123.123.123 npu_selid=ff dec_npuid=0 enc_npuid=0
proxyid=TESTVPN proto=0 sa=0 ref=1 serial=14
As seen above, 10.10.50.2 is not included as an allowed destination (AKA remote subnet) in the available traffic selector, and so it is correctly dropped as spoofed traffic.
Case 2: FortiGate sends initiating traffic over to the remote site, reply traffic arrives back on the FortiGate, and is dropped.
In this example, the client (10.180.244.217) at the FortiGate site is sending ICMP pings to a host at the remote site (10.190.9.27) over the AMA_CHRIS_GTD_2 VPN tunnel. Packet captures show the ICMP request leaving the FortiGate successfully and an ICMP reply being sent back over the VPN tunnel, but the ICMP reply is dropped by the FortiGate and not forwarded back out via LAN_CORE.
Notably, this VPN tunnel has multiple established Phase2 traffic selectors, one of which does exist to allow 10.180.244.0/24 -> 10.190.9.0/24, so this traffic should be allowed:
diagnose sniffer packet any 'host 10.180.244.217 and host 10.190.9.27 and icmp' 4
interfaces=[any]
filters=[host 10.180.244.217 and host 10.190.9.27 and icmp]
6.892227 LAN_CORE in 10.180.244.217 -> 10.190.9.27: icmp: echo request
6.892254 AMA_CHRIS_GTD_2 out 10.180.244.217 -> 10.190.9.27: icmp: echo request
7.045048 AMA_CHRIS_GTD_2 in 10.190.9.27 -> 10.180.244.217: icmp: echo reply
11.643004 LAN_CORE in 10.180.244.217 -> 10.190.9.27: icmp: echo request
11.643013 AMA_CHRIS_GTD_2 out 10.180.244.217 -> 10.190.9.27: icmp: echo request
11.795494 AMA_CHRIS_GTD_2 in 10.190.9.27 -> 10.180.244.217: icmp: echo reply
16.794303 AMA_CHRIS_GTD_2 in 10.190.9.27 -> 10.180.244.217: icmp: echo reply
Running the command get vpn ipsec tunnel summary repeatedly shows that the RX error counters are increasing for this VPN tunnel, even though other traffic streams are sending/receiving over the VPN seemingly without issue:
get vpn ipsec tunnel summary | grep AMA_CHRIS_GTD_2
'AMA_CHRIS_GTD_2' 52.71.79.65:0 selectors(total,up): 4/3 rx(pkt,err): 484/484 tx(pkt,err): 601/2
get vpn ipsec tunnel summary | grep AMA_CHRIS_GTD_2
'AMA_CHRIS_GTD_2' 52.71.79.65:0 selectors(total,up): 4/3 rx(pkt,err): 818/818 tx(pkt,err): 904/2
Running the debug flow shows that packets in the LAN_CORE -> AMA_CHRIS_GTD_2 direction are flowing correctly, but reply packets coming inbound on the VPN tunnel are being dropped due to anti-spoof check:
diagnose debug flow filter addr 10.180.244.217 10.190.9.27 and
diagnose debug flow show function-name enable
diagnose debug flow show iprope enable
diagnose debug flow trace start 1000
diagnose debug enable
Client to Server (10.180.244.217 -> 10.190.9.27):
func=print_pkt_detail line=6005 msg="vd-root:0 received a packet(proto=1, 10.180.244.217:1->10.190.9.27:2048) tun_id=0.0.0.0 from LAN_CORE. type=8, code=0, id=1, seq=18979."
func=init_ip_session_common line=6204 msg="allocate a new session-16eab499"
func=iprope_dnat_check line=5481 msg="in-[LAN_CORE], out-[]"
func=iprope_dnat_tree_check line=824 msg="len=0"
func=iprope_dnat_check line=5506 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
func=__vf_ip_route_input_rcu line=1989 msg="find a route: flag=00000000 gw-52.71.79.65 via AMA_CHRIS_GTD_2"
func=__iprope_fwd_check line=810 msg="in-[LAN_CORE], out-[AMA_CHRIS_GTD_2], skb_flags-02000000, vid-0, app_id: 0, url_cat_id: 0"
func=__iprope_tree_check line=524 msg="gnum-100004, use int hash, slot=4, len=10"
func=__iprope_user_identity_check line=1903 msg="ret-matched"
func=__iprope_check line=2404 msg="gnum-4e20, check-ffffffffa002cf60"
func=__iprope_check line=2421 msg="gnum-4e20 check result: ret-no-match, act-accept, flag-00000000, flag2-00000000"
func=__iprope_check_one_policy line=2374 msg="policy-1218 is matched, act-accept"
func=__iprope_fwd_check line=847 msg="after iprope_captive_check(): is_captive-0, ret-matched, act-accept, idx-1218"
func=iprope_fwd_auth_check line=876 msg="after iprope_captive_check(): is_captive-0, ret-matched, act-accept, idx-1218"
func=iprope_shaping_check line=974 msg="in-[LAN_CORE], out-[AMA_CHRIS_GTD_2], skb_flags-02000000, vid-0"
func=__iprope_check line=2404 msg="gnum-100015, check-ffffffffa002c2f0"
func=__iprope_check line=2421 msg="gnum-100015 check result: ret-no-match, act-accept, flag-00000000, flag2-00000000"
func=iprope_policy_group_check line=4903 msg="after check: ret-no-match, act-accept, flag-00000000, flag2-00000000"
func=fw_forward_handler line=1002 msg="Allowed by Policy-1218:"
func=ip_session_confirm_final line=3179 msg="npu_state=0x40000, hook=4"
func=ipsecdev_hard_start_xmit line=662 msg="enter IPSec interface AMA_CHRIS_GTD_2, tun_id=0.0.0.0"
func=_do_ipsecdev_hard_start_xmit line=222 msg="output to IPSec tunnel AMA_CHRIS_GTD_2, tun_id=52.71.79.65, vrf 0"
func=esp_output4 line=917 msg="IPsec encrypt/auth"
func=nipsec_set_ipsec_sa_enc line=934 msg="Trying to offload IPsec encrypt SA (p1/p2/spi={AMA_CHRIS_GTD_2/AMA_CHRIS_GTD_2_6/0xc47154a0}), npudev=1,
skb-dev=INTERNET_GTD"
func=ipsec_output_finish line=676 msg="send to 190.153.209.145 via intf-INTERNET_GTD"
Server to Client (10.190.9.27 -> 10.180.244.217):
id=65308 trace_id=102 func=print_pkt_detail line=6005 msg="vd-root:0 received a packet(proto=1, 10.190.9.27:1->10.180.244.217:0) tun_id=52.71.79.65 from AMA_CHRIS_GTD_2.
type=0, code=0, id=1, seq=18979."
id=65308 trace_id=102 func=resolve_ip_tuple_fast line=6107 msg="Find an existing session, id-16eab499, reply direction"
id=65308 trace_id=102 func=ipsec_spoofed4 line=243 msg="src ip 10.190.9.27 mismatch selector 0 range 10.155.0.0-10.155.255.255"
id=65308 trace_id=102 func=ipsec_input4 line=287 msg="anti-spoof check failed, drop"
As noted earlier, 10.180.244.0/24 <-> 10.190.9.0/24 should be allowed since a traffic selector does exist for this pairing. However, the debug flow shows that the ICMP reply traffic for 10.190.9.27 back to 10.180.244.217 was using a different traffic selector (10.155.0.0/16 <-> 10.190.9.0/24), and so it was dropped by the FortiGate due to anti-spoof check.
The reason this occurs is that multiple traffic selectors were available, and the remote VPN peer had sent traffic using the wrong traffic selector. One way to verify that this is happening is to run packet captures for ESP IPsec traffic between the FortiGate and the remote VPN peer, as the SPI value will be visible and unencrypted on each ESP packet. This value can be compared against the SPI value found under diagnose vpn tunnel list to confirm which Security Association (i.e., SPI and encrypt/decrypt keys) was used by the remote VPN device when transmitting the packet.
Solutions and workarounds:
As a temporary workaround, restarting the problematic VPN tunnel using diagnose vpn ike gateway clear name <tunnel_name> can be helpful for realigning both ends of the VPN and getting the correct Security Associations and Traffic Selectors to be used. However, if this becomes an ongoing issue (i.e., the remote VPN peer frequently uses the wrong SAs), then the solution is to reduce the number of negotiated traffic selectors, ideally down to a single selector.
On the FortiGate, it is generally recommended to use 0.0.0.0/0 as the local and remote subnet for the Phase2 selector, as this means that only a single traffic selector is required to service any source/destination traffic combo. However, the remote VPN gateway must be capable of configuring 0.0.0.0/0 as well to match the FortiGate (some VPN gateways are not capable of this).
Important: Setting Phase2 selectors to 0.0.0.0/0 is safe to do on the FortiGate, as administrators can and should restrict access across the VPN tunnel using a combination of Firewall Policies and network routes (there is very little need to use specific Phase2 selectors unless no other option exists).
As an alternative, consider combining multiple Phase2 selectors where possible to reduce the total number of selectors and therefore the likelihood of issues caused by remote VPN hosts.
Example: Combining Phase2 Selectors:
Consider the following three existing Phase2 selectors and their CLI equivalents:
- Phase2-1 : Source address : 10.10.0.0/24, Remote address : 192.168.10.0/24
- Phase2-2 : Source address : 10.10.0.0/24, Remote address : 192.168.20.0/24
- Phase2-3 : Source address : 10.10.0.0/24, Remote address : 192.168.30.0/24
CLI:
config vpn ipsec phase2-interface
edit "Phase2-1"
...
set src-subnet 10.10.0.0 255.255.255.0
set dst-subnet 192.168.10.0 255.255.255.0
next
edit "Phase2-2"
...
set src-subnet 10.10.0.0 255.255.255.0
set dst-subnet 192.168.20.0 255.255.255.0
next
edit "Phase2-3"
...
set src-subnet 10.10.0.0 255.255.255.0
set dst-subnet 192.168.30.0 255.255.255.0
next
end
These three selectors can be combined into a single selector, which is a pair of supernets that include these smaller subnets. For example, 192.168.0.0/19 includes 192.168.0.1-192.168.31.254, which would be the narrowest possible match for the original three subnets:
Phase2-1: Source address: 10.10.0.0/24, Remote address: 192.168.0.0/19.
CLI:
config vpn ipsec phase2-interface
edit "Phase2-1"
...
set src-subnet 10.10.0.0 255.255.255.0
set dst-subnet 192.168.0.0 255.255.224.0
next
end
As a simpler alternative, upsizing the remote subnet to 192.168.0.0/16 is much more straightforward and also allows easy future additions (i.e., it would allow 192.168.40.0/24, 192.168.50.0/24, etc., if added in the future).
Phase2-1: Source address: 10.10.0.0/24, Remote address: 192.168.0.0/16 (This would allow more subnets on the 3rd octet to be covered).
CLI:
config vpn ipsec phase2-interface
edit "Phase2-1"
...
set src-subnet 10.10.0.0 255.255.255.0
set dst-subnet 192.168.0.0 255.255.0.0
next
end
Note:
The issue is also experienced in versions after 7.2.11. A fix will be introduced in FortiOS v7.6.7 and 7.4.12 in order to allow traffic to pass as long as the source IP matches any selector on the tunnel.
Workaround for Mikrotik:
On the Action tab of the IPsec Policy, set Level to 'unique' instead of 'require'.
Related documents: