IPSEC VPN Issues

[ITACCCUT]

I am having issues with my IPSEC VPN not working. What I have boiled it down to is, it looks like when I try to send a ping from my computer to the firewall (B) using the internal interface IP on the the other side of the VPN. The firewall (A) I am behind does not forward the packet to the wan interface.
here is the flow trace filtered to the internal address of firewall (B). from the CLI of firewall (A)

The VPN appears to randomly stop working. Sometimes after a while it works.

id=20085 trace_id=25 func=print_pkt_detail line=5617 msg="vd-root:0 received a packet(proto=1, 10.0.0.83:1->10.2.0.254:2048) from internal. type=8, code=0, id=1, seq=18205."

id=20085 trace_id=25 func=init_ip_session_common line=5787 msg="allocate a new session-1d680973"

id=20085 trace_id=25 func=vf_ip_route_input_common line=2595 msg="find a route: flag=04000000 gw-10.2.0.254 via StG"

id=20085 trace_id=25 func=fw_forward_handler line=777 msg="Allowed by Policy-37:"

id=20085 trace_id=25 func=ipsecdev_hard_start_xmit line=788 msg="enter IPsec interface-StG"

id=20085 trace_id=25 func=esp_output4 line=927 msg="IPsec encrypt/auth"

id=20085 trace_id=25 func=ipsec_output_finish line=617 msg="send to 96.77.184.62 via intf-wan1"

Here is the flow trace to the external address for firewall (B) from the CLI of firewall (A). You can see that a ping directly to the external IP of Firewall (B) from a device behind firewall (A) works, but no Packets from the VPN appear to be exiting the WAN.

id=20085 trace_id=21 func=print_pkt_detail line=5617 msg="vd-root:0 received a packet(proto=1, 10.0.0.83:1->firewall B) from internal. type=8, code=0, id=1, seq=18200."

id=20085 trace_id=21 func=init_ip_session_common line=5787 msg="allocate a new session-1d67e3b6"

id=20085 trace_id=21 func=vf_ip_route_input_common line=2580 msg="Match policy routing id=2133131269: to 74.211.37.219 via ifindex-6"

id=20085 trace_id=21 func=vf_ip_route_input_common line=2595 msg="find a route: flag=04000000 gw-204.228.147.213 via wan2"

id=20085 trace_id=21 func=fw_forward_handler line=777 msg="Allowed by Policy-1: SNAT"

id=20085 trace_id=21 func=ids_receive line=289 msg="send to ips"

id=20085 trace_id=21 func=__ip_session_run_tuple line=3393 msg="SNAT 10.0.0.83->firewall A:60417"

id=20085 trace_id=22 func=print_pkt_detail line=5617 msg="vd-root:0 received a packet(proto=1, firewall B->firewall A:0) from wan2. type=0, code=0, id=60417, seq=18200."

id=20085 trace_id=22 func=resolve_ip_tuple_fast line=5697 msg="Find an existing session, id-1d67e3b6, reply direction"

id=20085 trace_id=22 func=__ip_session_run_tuple line=3407 msg="DNAT firewall A:0->10.0.0.83:1"

id=20085 trace_id=22 func=vf_ip_route_input_common line=2595 msg="find a route: flag=00000000 gw-10.0.0.83 via internal"

id=20085 trace_id=22 func=npu_handle_session44 line=1159 msg="Trying to offloading session from wan2 to internal, skb.npu_flag=00000000 ses.state=00012284 ses.npu_state=0x00001008"

id=20085 trace_id=22 func=fw_forward_dirty_handler line=399 msg="state=00012284, state2=00014001, npu_state=00001008"

id=20085 trace_id=22 func=ids_receive line=289 msg="send to ips"

Here is the flow filtered with the WAN address from firewall (A) in the CLI of firewall (B)

id=20085 trace_id=16 func=print_pkt_detail line=5824 msg="vd-root:0 received a packet(proto=1, Firewall (A):60417->Firewall (B):2048) from wan2. type=8, code=0, id=60417, seq=18206."
id=20085 trace_id=16 func=init_ip_session_common line=5995 msg="allocate a new session-00046253"
id=20085 trace_id=16 func=vf_ip_route_input_common line=2615 msg="find a route: flag=80000000 gw-74.211.37.219 via root"
id=20085 trace_id=17 func=print_pkt_detail line=5824 msg="vd-root:0 received a packet(proto=1, Firewall (B):60417->Firewall (A):0) from local. type=0, code=0, id=60417, seq=18206."
id=20085 trace_id=17 func=resolve_ip_tuple_fast line=5905 msg="Find an existing session, id-00046253, reply direction"
id=20085 trace_id=17 func=ipd_post_route_handler line=490 msg="out wan2 vwl_zone_id 0, state2 0x0, quality 0.
"

[Anthony_E]

Hello ITACCCUT,

Thank you for using the Community Forum.

I will seek to get you an answer or help. We will reply to this thread with an update as soon as possible.

Regards,

[syordanov]

Dear ITACCCUT ,

For the ICMP from 10.0.0.83 to 10.2.0.254 i can see that FW is matching FW policy rule No37 and traffic is forwarded to IPsec interface StG.
To check if traffic is leaving your device, please runn a sniffer like this bellow :

# diagnose sniffer packet any "host 10.0.0.83 and host 10.2.0.254 " 4

Also check if 10.0.0.83 and 10.2.0.254 are part of local and remove encryption domains of both FW's , also check the routing and if that traffic is received on remote VPN peer.

To check phase-1/phase-2 of your FW you can run the following :

# diagnose vpn ike gateway list name NAME_OF_VPN
# diagnose vpn tunnel list name NAME_OF_VPN

For the second debug flow ICMP traffic from 10.0.0.83 to firewall B , FW is matching policy routing No 2133131269, traffic is forwarded to wan2 and is allowed by SNAT policy , this i do not think is encrypted traffic .

[ITACCCUT]

Thank you for getting back to me. I will try out those suggestions and I will get back to you.

The bad part is this issue will not pop up until the internet has a hiccup and when it does it takes hours for the firewall to start functioning.

[ITACCCUT]

So I had the error again today.
This is what I found. Following your help and this article "https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-IPsec-VPNs-tunnels/ta-p/195955"
I ran this #diagnose vpn tunnel list name StG

list ipsec tunnel by names in vd 0
------------------------------------------------------
name=StG ver=1 serial=3 Firewall A:0->Firewall V:0 dst_mtu=1500
bound_if=6 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/520 options[0208]=npu frag-rfc run_state=0 accept_traffic=1 overlay_id=0

proxyid_num=1 child_num=0 refcnt=17 ilast=0 olast=0 ad=/0
stat: rxp=4921962 txp=8165279 rxb=1980619262 txb=7051953055
dpd: mode=on-demand on=1 idle=20000ms retry=3 count=0 seqno=1595
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=StG proto=0 sa=1 ref=218 serial=1
src: 0:10.0.0.0/255.255.255.0:0
dst: 0:10.2.0.0/255.255.255.0:0
SA: ref=6 options=10226 type=00 soft=0 mtu=1438 expire=40911/0B replaywin=2048
seqno=25d0 esn=0 replaywin_lastseq=00001db8 itn=0 qat=0 hash_search_len=1
life: type=01 bytes=0/0 timeout=42928/43200
dec: spi=97cc0151 esp=aes key=16 cca1f4fde7f8800eca6c2d20e98e687e
ah=sha1 key=20 8675c09b23558b6d8f8c7c1384927686d7d51578
enc: spi=8ec21637 esp=aes key=16 19e5893181615ad3888dbd744eb81d7d
ah=sha1 key=20 a5735ad9ea80a6727df5a8dd30d06769473c9f88
dec:pkts/bytes=9780/1617556, enc:pkts/bytes=16488/2091235
npu_flag=03 npu_rgwy=Firewall B npu_lgwy=Firewall A npu_selid=1 dec_npuid=1 enc_npuid=1
run_tally=1

And this #diagnose vpn ike gateway list name StG

vd: root/0
name: StG
version: 1
interface: wan2 6
addr: Firewall A:4500 -> Firewall B:4500
created: 366113s ago
IKE SA: created 1/5 established 1/5 time 140/4360/21220 ms
IPsec SA: created 1/9 established 1/9 time 60/62/70 ms

id/spi: 104 0f197b81125e2a62/b6f6456e59f25f62
direction: initiator
status: established 21691-21691s ago = 150ms
proposal: aes128-sha256
key: 5b120b2c02ec6f4d-65d19be8902d3cfb
lifetime/rekey: 86400/64408
DPD sent/recv: 0000063b/0000119c

When I ran this #diagnose sniffer packet any 'host Firewall B and port 4500' 4 0 l

I got little to no packets being sent or received.

[fluthersmack]

I'll give the recommendations a shot and report back.
The problem is that this won't become apparent until there's a hiccup in the internet, and even then it could be hours before the firewall begins protecting the network again.

                                                                                                                                                                                                                                                                     2048