FortiGate IPSec - wrong interface detected for incoming traffic

[Satory]

We have an IPSec tunnel between two FortiGate devices - FG500E and FG40F, both running version 7.0.14.

The IPSec is established without any problems, but the traffic inside the tunnel has some very strange issue. The tunnel IP addresses are 10.0.66.16/32 and 10.0.66.17/32.

The FG500E device sends the packets inside the tunnel, but when it receives the response, for example ping requests, it sees the traffic as received from the VLAN interface on which is built the tunnel, thus discarding the traffic. As a result the two tunnel interface ends cannot ping each other and the communication is not possible, as we use iBGP for routing.

Has anyone experienced some similar issue and how to fix this?

[saneeshpv_FTNT]

@Satory 

Could you please share your configuration related to IPSec, Firewall policy and routing at each end. Also if you have any capture showing it received on VLAN instead of the tunnel interface, please share here.

Yoy may also look into the below link mentioning "T to use dynamic routing with the tunnel or be able to ping the tunnel interface, specify an address for the remote end of the tunnel in remote-ip and an address for this end of the tunnel in IP. This is only available if the type is tunnel."

https://community.fortinet.com/t5/FortiGate/Technical-Note-Dynamic-routing-BGP-over-IPsec-tunnel/ta-...

Best Regards,

[Satory]

The setup of the IPSec and the interface on the core FortiGate is:

config vpn ipsec phase1-interface
    edit "O-BLA-DIS-PRIM"
        set interface "MAN_A1"
        set ike-version 2
        set local-gw X.X.X.X
        set peertype any
        set net-device disable
        set proposal aes256-sha512
        set dhgrp 21
        set nattraversal disable
        set remote-gw Y.Y.Y.Y
    next
end

config vpn ipsec phase2-interface
    edit "O-BLA-DIS-PRIM"
        set phase1name "O-BLA-DIS-PRIM"
        set proposal aes256-sha512
        set dhgrp 21
        set auto-negotiate enable
        set keylifeseconds 3600
    next
end

config system interface
    edit "O-BLA-DIS-PRIM"
        set vdom "root"
        set ip 10.0.66.16 255.255.255.255
        set allowaccess ping
        set type tunnel
        set remote-ip 10.0.66.17 255.255.255.255
        set role wan
        set snmp-index 93
        set interface "MAN_A1"
    next
end

On the other side the setup is analogical.

What I see when I ping from the remote end of the tunnel:

# diag sni pack any "host 10.0.66.17" 4 0 l
interfaces=[any]
filters=[host 10.0.66.17]
2024-02-22 14:36:21.118915 MAN_A1 in 10.0.66.17 -> 10.0.66.16: icmp: echo request
2024-02-22 14:36:22.130344 MAN_A1 in 10.0.66.17 -> 10.0.66.16: icmp: echo request
2024-02-22 14:36:23.140547 MAN_A1 in 10.0.66.17 -> 10.0.66.16: icmp: echo request
2024-02-22 14:36:24.150576 MAN_A1 in 10.0.66.17 -> 10.0.66.16: icmp: echo request
2024-02-22 14:36:25.160616 MAN_A1 in 10.0.66.17 -> 10.0.66.16: icmp: echo request

The above is as 100% spoofing...

[hbac]

Hi @Satory,

Please run the following debugs and try pinging again:

di deb disable
di deb res
diagnose debug flow filter clear
di deb flow filter addr 10.0.66.16
di deb flow filter proto 1
diagnose debug flow show function-name enable
di deb flow show iprope en
diagnose debug console timestamp enable
diagnose debug flow trace start 500
diagnose debug enable

Are you using 10.0.66.0 subnet on any other interfaces? You can run 'show full | grep 10.0.66 -f' to verify. 

Regards, 

[Satory]

The debug info is:

2024-02-22 17:15:42 id=20085 trace_id=1 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.0.66.17:1792->10.0.66.16:2048) tun_id=10.0.0.7 from MAN_A1. type=8, code=0, id=1792, seq=0."
2024-02-22 17:15:42 id=20085 trace_id=1 func=init_ip_session_common line=6046 msg="allocate a new session-1c295641, tun_id=10.0.0.7"
2024-02-22 17:15:42 id=20085 trace_id=1 func=iprope_dnat_check line=5336 msg="in-[MAN_A1], out-[]"
2024-02-22 17:15:42 id=20085 trace_id=1 func=iprope_dnat_tree_check line=827 msg="len=0"
2024-02-22 17:15:42 id=20085 trace_id=1 func=iprope_dnat_check line=5348 msg="result: skb_flags-02000008, vid-0, ret-no-match, act-accept, flag-00000000"
2024-02-22 17:15:42 id=20085 trace_id=1 func=ip_route_input_slow line=2267 msg="reverse path check fail, drop"
2024-02-22 17:15:42 id=20085 trace_id=1 func=ip_session_handle_no_dst line=6132 msg="trace"
2024-02-22 17:15:43 id=20085 trace_id=2 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.0.66.17:1792->10.0.66.16:2048) tun_id=10.0.0.7 from MAN_A1. type=8, code=0, id=1792, seq=1."
2024-02-22 17:15:43 id=20085 trace_id=2 func=init_ip_session_common line=6046 msg="allocate a new session-1c29580f, tun_id=10.0.0.7"
2024-02-22 17:15:43 id=20085 trace_id=2 func=iprope_dnat_check line=5336 msg="in-[MAN_A1], out-[]"
2024-02-22 17:15:43 id=20085 trace_id=2 func=iprope_dnat_tree_check line=827 msg="len=0"
2024-02-22 17:15:43 id=20085 trace_id=2 func=iprope_dnat_check line=5348 msg="result: skb_flags-02000008, vid-0, ret-no-match, act-accept, flag-00000000"
2024-02-22 17:15:43 id=20085 trace_id=2 func=ip_route_input_slow line=2267 msg="reverse path check fail, drop"
2024-02-22 17:15:43 id=20085 trace_id=2 func=ip_session_handle_no_dst line=6132 msg="trace"
2024-02-22 17:15:44 id=20085 trace_id=3 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.0.66.17:1792->10.0.66.16:2048) tun_id=10.0.0.7 from MAN_A1. type=8, code=0, id=1792, seq=2."
2024-02-22 17:15:44 id=20085 trace_id=3 func=init_ip_session_common line=6046 msg="allocate a new session-1c2959af, tun_id=10.0.0.7"
2024-02-22 17:15:44 id=20085 trace_id=3 func=iprope_dnat_check line=5336 msg="in-[MAN_A1], out-[]"
2024-02-22 17:15:44 id=20085 trace_id=3 func=iprope_dnat_tree_check line=827 msg="len=0"
2024-02-22 17:15:44 id=20085 trace_id=3 func=iprope_dnat_check line=5348 msg="result: skb_flags-02000008, vid-0, ret-no-match, act-accept, flag-00000000"
2024-02-22 17:15:44 id=20085 trace_id=3 func=ip_route_input_slow line=2267 msg="reverse path check fail, drop"
2024-02-22 17:15:44 id=20085 trace_id=3 func=ip_session_handle_no_dst line=6132 msg="trace"
2024-02-22 17:15:45 id=20085 trace_id=4 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.0.66.17:1792->10.0.66.16:2048) tun_id=10.0.0.7 from MAN_A1. type=8, code=0, id=1792, seq=3."
2024-02-22 17:15:45 id=20085 trace_id=4 func=init_ip_session_common line=6046 msg="allocate a new session-1c295c11, tun_id=10.0.0.7"
2024-02-22 17:15:45 id=20085 trace_id=4 func=iprope_dnat_check line=5336 msg="in-[MAN_A1], out-[]"
2024-02-22 17:15:45 id=20085 trace_id=4 func=iprope_dnat_tree_check line=827 msg="len=0"
2024-02-22 17:15:45 id=20085 trace_id=4 func=iprope_dnat_check line=5348 msg="result: skb_flags-02000008, vid-0, ret-no-match, act-accept, flag-00000000"
2024-02-22 17:15:45 id=20085 trace_id=4 func=ip_route_input_slow line=2267 msg="reverse path check fail, drop"
2024-02-22 17:15:45 id=20085 trace_id=4 func=ip_session_handle_no_dst line=6132 msg="trace"
2024-02-22 17:15:46 id=20085 trace_id=5 func=print_pkt_detail line=5867 msg="vd-root:0 received a packet(proto=1, 10.0.66.17:1792->10.0.66.16:2048) tun_id=10.0.0.7 from MAN_A1. type=8, code=0, id=1792, seq=4."
2024-02-22 17:15:46 id=20085 trace_id=5 func=init_ip_session_common line=6046 msg="allocate a new session-1c295e96, tun_id=10.0.0.7"
2024-02-22 17:15:46 id=20085 trace_id=5 func=iprope_dnat_check line=5336 msg="in-[MAN_A1], out-[]"
2024-02-22 17:15:46 id=20085 trace_id=5 func=iprope_dnat_tree_check line=827 msg="len=0"
2024-02-22 17:15:46 id=20085 trace_id=5 func=iprope_dnat_check line=5348 msg="result: skb_flags-02000008, vid-0, ret-no-match, act-accept, flag-00000000"
2024-02-22 17:15:46 id=20085 trace_id=5 func=ip_route_input_slow line=2267 msg="reverse path check fail, drop"
2024-02-22 17:15:46 id=20085 trace_id=5 func=ip_session_handle_no_dst line=6132 msg="trace"

The check is OK - only finds this interface.

[hbac]

@Satory,

That's weird. Instead of pinging from the tunnel interface, can you ping from internal an internal host behind the firewall? Also, can you disable npu-offload, bounce the tunnel, and try again. 

config vpn ipsec phase1-interface 

edit <>

set npu-offload dis

end 

Regards, 

[Satory]

Setting it to disabled npu offload did not solved the problem!

[saneeshpv_FTNT]

@Satory 

Do you see the packet being encrypted from the remote side of the tunnel when you ping the destination 10.0.66.16 and also are you seeing any ESP packets being forwarded from the remote side physical interface. 

You can also verify if you see any ESP packets being received at your local side (on MAN_A1) when the ping is made from remote end. Also please share your routing information from both sides as I don't see any phase 2 selectors defined for interesting traffic. 

You can also try to ping internal network as @hbac mentioned to see if this works well.

You need to make sure you have proper route to push the traffic to the tunnel interface at each end.

Provide below outputs as well.

diagnose vpn ike gateway list name <phase1name>

diagnose vpn tunnel list name <phase2name>

Best Regards

[Satory]

Yes, from the remote side I see all going as ESP encrypted.

On the HUB side tho I do not see it that way, but taking into account I see the encrypted data I believe it is part of the issue.

The diagnose data:

# diagnose vpn ike gateway list name O-BLA-DIS-PRIM

vd: root/0
name: O-BLA-DIS-PRIM
version: 2
interface: MAN_A1 50
addr: 192.168.3.3:500 -> 192.168.2.147:500
tun_id: 10.0.0.7/::10.0.0.7
remote_location: 0.0.0.0
network-id: 0
virtual-interface-addr: 10.0.66.16 -> 0.0.0.0
created: 195s ago
peer-id: 192.168.2.147
peer-id-auth: no
PPK: no
IKE SA: created 1/2  established 1/2  time 30/10535/21040 ms
IPsec SA: created 1/2  established 1/2  time 0/10520/21040 ms

  id/spi: 1734544 3a663e0a3a927909/25e5c9f53118cfb1
  direction: initiator
  status: established 195-174s ago = 21040ms
  proposal: aes256-sha512
  child: no
  SK_ei: bc279ea03284ae83-e8adcc77d03d5ac4-43e5f77527f9279d-2cc3fa1621bd17c7
  SK_er: fa8284a838fef262-2cdf77b066244b4d-5e0ee958cb98bfb1-27831d7428ce50db
  SK_ai: 2dc1879c255e62ae-09a58291c2b9a037-2b6a4ec62c111a4e-7548df4975259beb-bffe10e0af407724-d89731d66b88c5bc-70d1c05c5601e365-fa755da15ccb9452
  SK_ar: 491ed2156e234974-51ea2870f8ef7125-69d45014d70ef093-454710b24360ca04-50746e077dd88023-a7d793e9b0b493b2-56ddd47fafb13aa0-2bc3705e5af0c3f7
  PPK: no
  message-id sent/recv: 2/3
  lifetime/rekey: 86400/85925
  DPD sent/recv: 00000000/00000000
  peer-id: 192.168.2.147





# diagnose vpn tunnel list name O-BLA-DIS-PRIM
list ipsec tunnel by names in vd 0
------------------------------------------------------
name=O-BLA-DIS-PRIM ver=2 serial=90 192.168.3.3:0->192.168.2.147:0 tun_id=10.0.0.7 tun_id6=::10.0.0.7 dst_mtu=1500 dpd-link=on weight=1
bound_if=50 lgwy=static/1 tun=intf mode=auto/1 encap=none/544 options[0220]=frag-rfc  run_state=0 role=sync-primary accept_traffic=1 overlay_id=0

proxyid_num=1 child_num=0 refcnt=5 ilast=124 olast=44031605 ad=/0
stat: rxp=11 txp=0 rxb=780 txb=0
dpd: mode=on-demand on=1 idle=20000ms retry=3 count=0 seqno=0
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=O-BLA-DIS-PRIM proto=0 sa=1 ref=2 serial=2 auto-negotiate
  src: 0:0.0.0.0-255.255.255.255:0
  dst: 0:0.0.0.0-255.255.255.255:0
  SA:  ref=3 options=18227 type=00 soft=0 mtu=1422 expire=3146/0B replaywin=2048
       seqno=1 esn=0 replaywin_lastseq=00000009 qat=0 rekey=0 hash_search_len=1
  life: type=01 bytes=0/0 timeout=3303/3600
  dec: spi=e61f6ceb esp=aes key=32 50e616218d9eae5498afbcf87adb26bac2194077f601815b8ac1a4c51dd1f174
       ah=sha512 key=64 77279f60e4a379718c029ffc4719528b0014d85bd32c33aacbaf5a46264305a48e5ac39d5c332c386dc8dea0875ceeec818b7fea9b939d7be3082295b6db2f32
  enc: spi=8edcf5cc esp=aes key=32 0aac49179693c00c0024ef96ff0c5a5e4873ce0ae9813df13d54d35f4c0fb9c4
       ah=sha512 key=64 dd409ed0b0cde9a2372dc97c649f5fc93365613bc02a66770fb891c1544bbfeaa1c1d35ec8615427163d28e5d809f8ddb30acf1572317022fc0bc8bf24bda734
  dec:pkts/bytes=16/1056, enc:pkts/bytes=0/0
  npu_flag=00 npu_rgwy=192.168.2.147 npu_lgwy=192.168.3.3 npu_selid=ac dec_npuid=0 enc_npuid=0
run_tally=0

[saneeshpv_FTNT]

@Satory 

Try to make changes as per article as below and let me know the results.

https://community.fortinet.com/t5/FortiGate/Technical-Tip-How-to-use-exchange-interface-IP-in-IPsec-...