| Description |
This article describes an condition that can prevent site-to-site IPsec traffic when a local-in policy on one side is misconfigured to block legitimate port 500 or 4500 traffic.
In this situation, the tunnel initially comes up and can pass traffic, but after some time one side of the IPsec tunnel may go down while the other IPsec gateway shows the tunnel as still being up. |
| Scope |
FortiGate, IPsec site-to-site VPN. |
| Solution |
Topology:
Configuration: An IPsec site-to-site VPN is configured on both sides with basic settings created by 'IPsec Wizard' as shown in Technical Tip: How to configure VPN Site to Site between FortiGates (Using VPN Setup Wizard). Part of the IPsec phase 1 configuration is shown below. The Dead Peer Detection (DPD) setting is 'on-demand' by default.
Site 1:
config vpn ipsec phase1-interface edit "site1" set interface "port1" set peertype any set net-device disable set proposal aes128-sha256 aes256-sha256 aes128-sha1 aes256-sha1 set comments "VPN: site1 (Created by VPN wizard)" set wizard-type static-fortigate set remote-gw 10.56.240.221 set psksecret ENC next end
Site 2:
config vpn ipsec phase1-interface edit "site2" set interface "port1" set peertype any set net-device disable set proposal aes128-sha256 aes256-sha256 aes128-sha1 aes256-sha1 set comments "VPN: site2 (Created by VPN wizard)" set wizard-type static-fortigate set remote-gw 10.56.240.208 set psksecret ENC next end
On Site 1, no local-in policy is configured, and on Site 2, there are some local-in policies configured to limit to only certain types of connection to the Site 2 FortiGate. The 'IKE' service is not included in the 'accept' policy 1 so it will be denied by policy 2 (implicit deny).
Local-in policies on Site 2:
config firewall local-in-policy edit 1 set intf "port1" set srcaddr "office1" "office2" set dstaddr "all" set action accept set service "HTTPS" set schedule "always" set comments "allow mgmt from offices" next edit 2 set intf "port2" set srcaddr "all" set dstaddr "all" set service "ALL" set schedule "always" set comments "implicit deny" next end
The 'IKE' service object is includes ports UDP 500 & 4500. If these are not allowed, inbound IKE initiation requests will be denied. However, the tunnel still comes up initially since the IKE initiation from Site 2 is allowed by Site 1 and since a site-to-site tunnel negotiation can be initiated from either side.
The session list on Site 2 indicates a local traffic session has been created for the local UDP 500 packet. As long as this session is present in the table, matching IKE traffic from Site 1 is not dropped by Site 2 local-in policy. This is by design since local-in policies only affect connections established from remote sources.
diagnose sys session filter dport 500 diagnose sys session list session info: proto=17 proto_state=01 duration=5 expire=177 timeout=0 flags=00000000 socktype=0 sockport=0 av_idx=0 use=3 origin-shaper= reply-shaper= per_ip_shaper= class_id=0 ha_id=0 policy_dir=0 tunnel=/ vlan_cos=255/255 state=local nds statistic(bytes/packets/allow_err): org=1756/7/1 reply=1616/4/1 tuples=2 tx speed(Bps/kbps): 325/2 rx speed(Bps/kbps): 299/2 orgin->sink: org out->post, reply pre->in dev=0->3/3->13 gwy=0.0.0.0/10.56.240.221 hook=out dir=org act=noop 10.56.240.221:500->10.56.240.208:500(0.0.0.0:0) hook=in dir=reply act=noop 10.56.240.208:500->10.56.240.221:500(0.0.0.0:0) dst_mac=00:63:33:70:17:01 misc=0 policy_id=0 pol_uuid_idx=0 auth_info=0 chk_client_info=0 vd=0 serial=005c7388 tos=ff/ff app_list=0 app=0 url_cat=0 rpdb_link_id=00000000 ngfwid=n/a npu_state=00000000 no_ofld_reason: local total session 1
Debug flow diagnostics show the traffic matching the existing session.
id=20085 trace_id=105 func=print_pkt_detail line=5844 msg="vd-root:0 received a packet(proto=17, 10.56.240.208:500->10.56.240.221:500) tun_id=0.0.0.0 from port1. " id=20085 trace_id=105 func=resolve_ip_tuple_fast line=5930 msg="Find an existing session, id-005c8c6b, reply direction" id=20085 trace_id=106 func=print_pkt_detail line=5844 msg="vd-root:0 received a packet(proto=17, 10.56.240.208:500->10.56.240.221:500) tun_id=0.0.0.0 from port1. " id=20085 trace_id=106 func=resolve_ip_tuple_fast line=5930 msg="Find an existing session, id-005c8c6b, reply direction"
In the IPsec phase 1 setting, the DPD is set to 'on-demand' by default, which means Site 1 will only send DPD messages when IPsec traffic is sent but no reply is received from the peer. In this topology, this setting applies to the scenario in the below graph. Site 2 does not send any DPD message since data traffic is being received over the tunnel.
After the local port 500 session expires on Site 2, any IKE packet from Site 1 will trigger a local-in policy lookup and be blocked by local-in policy 2.
c3po-kvm36 # id=20085 trace_id=101 func=print_pkt_detail line=5844 msg="vd-root:0 received a packet(proto=17, 10.56.240.208:500->10.56.240.221:500) tun_id=0.0.0.0 from port1. " id=20085 trace_id=101 func=init_ip_session_common line=6023 msg="allocate a new session-005c892d, tun_id=0.0.0.0" id=20085 trace_id=101 func=vf_ip_route_input_common line=2605 msg="find a route: flag=84000000 gw-10.56.240.221 via root" id=20085 trace_id=101 func=fw_local_in_handler line=500 msg="iprope_in_check() check failed on policy 2, drop"
ESP traffic to Site 2 will not be dropped by default even if not explicitly allowed by local-in policy, see Technical Tip: ESP traffic handling with respect to local-in policies on a FortiGate Firewall.
Note: The port 500 session of IKE traffic for Site 2 can only be re-created when that side sends an IKE packet. However, this will not occur unless a new auto-negotiation is triggered or Site 2 sends data traffic to Site 1 but sees no response to trigger DPD by Site 2.
Site 1 is sending DPD R-U-THERE messages but receives no R-U-THERE-ACK. After 3 consecutive DPD failures, the Site 1 FortiGate brings down the tunnel by deleting the IPsec SA, informs Site 2 FortiGate about the IPsec SA deletion, and brings down the IKE SA. The IKE debug shows the tunnel closure.
2024-01-14 16:17:01.904150 ike V=root:0:site1: link is idle 3 10.56.240.208->10.56.240.221:0 dpd=2 seqno=12 rr=0 2024-01-14 16:17:01.905104 ike V=root:0:site1:131: send IKEv1 DPD probe, seqno 12 2024-01-14 16:17:01.909535 ike V=root:0:site1:131: sent IKE msg (R-U-THERE): 2024-01-14 16:17:22.384248 ike V=root:0:site1: link is idle 3 10.56.240.208->10.56.240.221:0 dpd=2 seqno=12 rr=0 2024-01-14 16:17:22.385198 ike V=root:0:site1:131: send IKEv1 DPD probe, seqno 12 2024-01-14 16:17:22.389696 ike V=root:0:site1:131: sent IKE msg (R-U-THERE): 2024-01-14 16:17:42.864267 ike V=root:0:site1: link is idle 3 10.56.240.208->10.56.240.221:0 dpd=2 seqno=12 rr=0 2024-01-14 16:17:42.865261 ike V=root:0:site1:131: send IKEv1 DPD probe, seqno 12 2024-01-14 16:17:42.869840 ike V=root:0:site1:131: sent IKE msg (R-U-THERE): 10.56.240.208:500->10.56.240.221:500, len=108, vrf=0, id=e8bbbfcd882e5ac5/73e8daa9c4694eeb:86a68017 2024-01-14 16:18:03.344219 ike V=root:0:site1: link fail 3 10.56.240.208->10.56.240.221:0 dpd=2 2024-01-14 16:18:03.345034 ike V=root:0:site1: link down 3 10.56.240.208->10.56.240.221:500 2024-01-14 16:18:03.345776 ike V=root:0:site1: going to be deleted 2024-01-14 16:18:03.346470 ike V=root:0:site1: flushing 2024-01-14 16:18:03.347046 ike V=root:0:site1: deleting IPsec SA with SPI c736a9de 2024-01-14 16:18:03.347757 ike V=root:0:site1:site1: deleted IPsec SA with SPI c736a9de, SA count: 0 2024-01-14 16:18:03.348590 ike V=root:0:site1: sending SNMP tunnel DOWN trap for site1 2024-01-14 16:18:03.349329 ike V=root:0:site1:131: send IPsec SA delete, spi 6c947eb2 2024-01-14 16:18:03.353356 ike V=root:0:site1:131: sent IKE msg (IPsec SA_DELETE-NOTIFY): 10.56.240.208:500->10.56.240.221:500, len=92, vrf=0, id=e8bbbfcd882e5ac5/73e8daa9c4694eeb:ed27d45d 2024-01-14 16:18:03.355385 ike V=root:0:site1: flushed 2024-01-14 16:18:03.367961 ike V=root:0:site1: set oper down
However, this deletion notification (IPsec SA_DELETE-NOTIFY) is also dropped by local-in policy 2 on the Site 2 FortiGate with reaching the IKE daemon. For this reason, the IPsec tunnel will show as down on Site 1 but will remain showing up on Site 2.
After the IPsec tunnel on Site 1 goes down, it will start auto-negotiation. However, the IKE negotiation request will still be dropped by the local-in policy on Site 2. The IPsec tunnel on these two FortiGates will remain in this status until the tunnel on Site 2 is manually flushed to trigger a new IKE negotiation, or until the Site 2 FortiGate attempts to send any traffic over the tunnel to Site 1.
Resolution: If using local-in policies that would block some IKE traffic, configure a local-in-policy to explicitly allow IKE traffic from intended source addresses as demonstrated in the article Technical Tip: How to block Unwanted IKE Negotiations and unknown ESP/SPI Packets with Local-In Poli.... If IKE traffic is not allowed on both sides of the IPsec VPN tunnel, it could cause tunnel instability requiring manual intervention to clear.
Related articles: Technical Tip: Configuring DPD (dead peer detection) on IPsec VPN |
