Hub and Spoke IPSEC VPN to Meraki MX not passing traffic. Both Phase 1 and 2 are up.

[Trooper]

This is a lab environment that I am doing in prep for a production rollout.

The hub is a FortiGate 60F, version 7.0.2 Build 0234 (GA)

The spoke is a Meraki MX67, version 15.43.1

The IPSEC tunnel is a Dialup User, as the remote Meraki's will have unknown public IP addresses.

Both Phase 1 and 2 are up, but traffic is not passing both ways.

The issue looks like the FGT 60F is not sending the return traffic.

Hub site - 10.8.0.0/16

Remote - 10.11.7.0/24

FGT-60F WAN IP - 10.48.0.67

Meraki WAN IP - 10.48.0.104

Tunnel interface is called: EH

I am running a continuous ping (once a second) from a remote client (10.11.7.2) to a PC behind the FGT (the PC is 10.8.1.100).

Wireshark on the 10.8.1.100 client shows correct echo and echo replies.

Debugs on the FGT show the return traffic entering the tunnel interface.

But it never comes out at the Meraki end.

I suspect the FGT is not sending it properly.

I have Wireshark on the external interfaces, and I do not see the encapsulated traffic from the FGT.

(Except some kind of keepalive every 10 seconds)

I do see encapsulated traffic from the Meraki every one second.

I have Firewall policies to allow tunnel traffic in and out.

I have a static route pointing the remote subnet towards the tunnel interface.

I’ve reviewed some posts with similar issues but have not resolved mine.

Here are some debug outputs.

I see this: tun_id=10.0.0.1 in the first debug.

Not sure where 10.0.0.1 is coming from, or if it matters.

Thanks

FortiGate60F # diagnose vpn tunnel list
list all ipsec tunnel in vd 0
------------------------------------------------------
name=EH_0 ver=1 serial=3 10.48.0.67:4500->10.48.0.104:4500 tun_id=10.48.0.104 dst_mtu=0 dpd-link=on weight=1
bound_if=5 lgwy=static/1 tun=tunnel/255 mode=dial_inst/3 encap=none/9088 options[2380]=rgwy-chg rport-chg frag-rfc run_state=0 accept_traffic=1 overlay_id=0
parent=EH index=0
proxyid_num=1 child_num=0 refcnt=4 ilast=0 olast=0 ad=/0
stat: rxp=115 txp=0 rxb=14720 txb=0
dpd: mode=off on=0 idle=60000ms retry=3 count=0 seqno=0
natt: mode=silent draft=32 interval=10 remote_port=4500
proxyid=EH proto=0 sa=1 ref=2 serial=1
src: 0:10.8.0.0-10.8.255.255:0
dst: 0:10.11.7.0-10.11.7.255:0
SA: ref=3 options=24 type=00 soft=0 mtu=1280 expire=28497/0B replaywin=0
seqno=1 esn=0 replaywin_lastseq=00000000 itn=0 qat=0 hash_search_len=1
life: type=01 bytes=0/0 timeout=28788/28800
dec: spi=0ece1e9f esp=aes key=32 e62ee7db7d9b08ea142a5d8839f89fa1989be0cca5834e74762c233fb0b8d9d0
ah=sha1 key=20 dd7516336594f846409afde624b09d6207953875
enc: spi=caa07d59 esp=aes key=32 4f3880d62b1e7349b86aabadca4e7f87066942e0666b422efc5ee272e243c170
ah=sha1 key=20 4a594af9d5e10f8d1ee5865e280782d589e6db1d
dec:pkts/bytes=115/6900, enc:pkts/bytes=0/0
npu_flag=00 npu_rgwy=10.48.0.104 npu_lgwy=10.48.0.67 npu_selid=1 dec_npuid=0 enc_npuid=0
------------------------------------------------------
name=EH ver=1 serial=1 10.48.0.67:0->0.0.0.0:0 tun_id=10.0.0.1 dst_mtu=0 dpd-link=on weight=1
bound_if=5 lgwy=static/1 tun=tunnel/255 mode=dialup/2 encap=none/512 options[0200]=frag-rfc accept_traffic=1 overlay_id=0
proxyid_num=0 child_num=1 refcnt=3 ilast=2698 olast=2698 ad=/0
stat: rxp=1147 txp=0 rxb=146816 txb=0
dpd: mode=off on=0 idle=20000ms retry=3 count=0 seqno=0
natt: mode=none draft=0 interval=0 remote_port=0
run_tally=0

FortiGate60F # diagnose sniffer packet any icmp 4
interfaces=[any]
filters=[icmp]
2.089177 EH in 10.11.7.2 -> 10.8.1.100: icmp: echo request
2.089221 internal1 out 10.11.7.2 -> 10.8.1.100: icmp: echo request
2.089694 internal1 in 10.8.1.100 -> 10.11.7.2: icmp: echo reply
2.089711 EH out 10.8.1.100 -> 10.11.7.2: icmp: echo reply
4.085990 EH in 10.11.7.2 -> 10.8.1.100: icmp: echo request
4.086028 internal1 out 10.11.7.2 -> 10.8.1.100: icmp: echo request
4.086502 internal1 in 10.8.1.100 -> 10.11.7.2: icmp: echo reply
4.086519 EH out 10.8.1.100 -> 10.11.7.2: icmp: echo reply
6.084978 EH in 10.11.7.2 -> 10.8.1.100: icmp: echo request
6.085018 internal1 out 10.11.7.2 -> 10.8.1.100: icmp: echo request
6.085518 internal1 in 10.8.1.100 -> 10.11.7.2: icmp: echo reply
6.085535 EH out 10.8.1.100 -> 10.11.7.2: icmp: echo reply

FortiGate60F # diag debug flow show function-name enable
show function name
FortiGate60F # diag debug flow filter proto 1
FortiGate60F # diag debug flow filter addr 10.8.1.100
FortiGate60F # diag debug enable
FortiGate60F # diag debug flow trace start 10

FortiGate60F # id=20085 trace_id=1 func=print_pkt_detail line=5783 msg="vd-root:0 received a packet(proto=1, 10.11.7.2:1->10.8.1.100:2048) from EH. type=8, code=0, id=1, seq=8484."
id=20085 trace_id=1 func=resolve_ip_tuple_fast line=5864 msg="Find an existing session, id-00001a5d, original direction"
id=20085 trace_id=1 func=npu_handle_session44 line=1161 msg="Trying to offloading session from EH to internal1, skb.npu_flag=00000400 ses.state=00010204 ses.npu_state=0x06040000"
id=20085 trace_id=1 func=fw_forward_dirty_handler line=396 msg="state=00010204, state2=00000001, npu_state=06040000"
id=20085 trace_id=2 func=print_pkt_detail line=5783 msg="vd-root:0 received a packet(proto=1, 10.8.1.100:1->10.11.7.2:0) from internal1. type=0, code=0, id=1, seq=8484."
id=20085 trace_id=2 func=resolve_ip_tuple_fast line=5864 msg="Find an existing session, id-00001a5d, reply direction"
id=20085 trace_id=2 func=npu_handle_session44 line=1161 msg="Trying to offloading session from internal1 to EH, skb.npu_flag=00000400 ses.state=00000204 ses.npu_state=0x06040000"
id=20085 trace_id=2 func=fw_forward_dirty_handler line=396 msg="state=00000204, state2=00000001, npu_state=06040000"
id=20085 trace_id=2 func=ipsecdev_hard_start_xmit line=625 msg="enter IPSec interface EH"
id=20085 trace_id=3 func=print_pkt_detail line=5783 msg="vd-root:0 received a packet(proto=1, 10.11.7.2:1->10.8.1.100:2048) from EH. type=8, code=0, id=1, seq=8485."
id=20085 trace_id=3 func=resolve_ip_tuple_fast line=5864 msg="Find an existing session, id-00001a5d, original direction"
id=20085 trace_id=3 func=npu_handle_session44 line=1161 msg="Trying to offloading session from EH to internal1, skb.npu_flag=00000400 ses.state=00010204 ses.npu_state=0x06040000"
id=20085 trace_id=3 func=fw_forward_dirty_handler line=396 msg="state=00010204, state2=00000001, npu_state=06040000"
id=20085 trace_id=4 func=print_pkt_detail line=5783 msg="vd-root:0 received a packet(proto=1, 10.8.1.100:1->10.11.7.2:0) from internal1. type=0, code=0, id=1, seq=8485."
id=20085 trace_id=4 func=resolve_ip_tuple_fast line=5864 msg="Find an existing session, id-00001a5d, reply direction"
id=20085 trace_id=4 func=npu_handle_session44 line=1161 msg="Trying to offloading session from internal1 to EH, skb.npu_flag=00000400 ses.state=00000204 ses.npu_state=0x06040000"
id=20085 trace_id=4 func=fw_forward_dirty_handler line=396 msg="state=00000204, state2=00000001, npu_state=06040000"
id=20085 trace_id=4 func=ipsecdev_hard_start_xmit line=625 msg="enter IPSec interface EH"

[akristof]

Hi,

Thank you for your question. VPN looks OK, but I would try one thing. Under phase1-interface enable net-device and retest. The debug flow is missing something, not sure if it is trimmed. Ideal would be if you would have debug flow when session is established (before ping starts), usually that says the most useful info.

[Avensus]

Hi,

Have you been able to resolve this? We are experiencing the exact same issue, with tunnel (P1 & P2) being up and no traffic able to pass from both sides.