IPsec site to site, traffic come through wan instead of tunnel interface

[kyw74]

Hi, I have setup IPsec s2s vpn between two site, A and B

A is behind a NAT router,

topology:

192.168.63.0/24 <-> A (VPN router) <-> NAT router <-> internet <-> B (fortigate) <-> 192.168.2.0/24

I've successfully established phase 2 IPsec tunnels between A and B, but I can't ping hosts from A to B, or B to A.

After some troubleshooting, I found somethings really weird:

If I ping from B to A, traffic is pass through the tunnel interface (tun02)

forti01 # diagnose sniffer packet tun02
interfaces=[tun02]
filters=[none]
4.162893 192.168.2.1 -> 192.168.63.110: icmp: echo request
5.178048 192.168.2.1 -> 192.168.63.110: icmp: echo request
6.201902 192.168.2.1 -> 192.168.63.110: icmp: echo request
7.225765 192.168.2.1 -> 192.168.63.110: icmp: echo request

but if I ping from A to B, traffic is coming from wan interface (wan2) instead of tunnel interface, no traffic is coming through tunnel interface

forti01 # diagnose sniffer packet tun02

interfaces=[tun02]
filters=[none]

nothing shows at all

forti01 # diagnose sniffer packet wan2 'host 192.168.2.1 and icmp'
interfaces=[wan2]
filters=[host 192.168.2.1 and icmp]
0.370676 192.168.63.110 -> 192.168.2.1: icmp: echo request
1.371506 192.168.63.110 -> 192.168.2.1: icmp: echo request
2.374358 192.168.63.110 -> 192.168.2.1: icmp: echo request
3.376188 192.168.63.110 -> 192.168.2.1: icmp: echo request

I have other production ipsec s2s tunnels connected to other sites which work properly, both way traffic is pass through tunnel interface, not wan.

Also, static route from B to A looks wired as well, the static route gateway IP is 10.0.0.6 (created with firmware 7.0.6),  instead of A's public IP address. My other working IPsec tunnels, static route to target sites are all their public IP addresses (created with firmware 7.0.4).

Any hints for further troubleshooting?

[kyw74]

I've just do another test:

diag de flow filter addr 192.168.63.110

diag de flow filter proto 1

diag de flow trace start 100 

diag de en

then ping from 192.168.63.110 to 192.168.2.1

from the console, I see:

id=20085 trace_id=51 func=print_pkt_detail line=5863 msg="vd-root:0 received a packet(proto=1, 192.168.63.110:43279->192.168.2.1:2048) tun_id=10.0.0.6 from wan2. type=8, code=0, id=43279, seq=0."
id=20085 trace_id=51 func=init_ip_session_common line=6042 msg="allocate a new session-0f1ee2c6, tun_id=10.0.0.6"
id=20085 trace_id=51 func=vf_ip_route_input_common line=2605 msg="find a route: flag=00000000 gw-192.168.2.1 via port2"
id=20085 trace_id=51 func=fw_forward_handler line=717 msg="Denied by forward policy check (policy 0)"
id=20085 trace_id=52 func=print_pkt_detail line=5863 msg="vd-root:0 received a packet(proto=1, 192.168.63.110:43279->192.168.2.1:2048) tun_id=10.0.0.6 from wan2. type=8, code=0, id=43279, seq=1."
id=20085 trace_id=52 func=init_ip_session_common line=6042 msg="allocate a new session-0f1ee2d1, tun_id=10.0.0.6"
id=20085 trace_id=52 func=vf_ip_route_input_common line=2605 msg="find a route: flag=00000000 gw-192.168.2.1 via port2"
id=20085 trace_id=52 func=fw_forward_handler line=717 msg="Denied by forward policy check (policy 0)"

I do have a ACCEPT forward policy from "wan2 + tun02", to "port2", source "all", dst "all", service "ALL" defined, but it never get any hits.

[knaveenkumar]

Hi , 

1. please check if there is any policy route 

2.Aslo can you please check distance and priority of static route 

3 # get router info routing table all

4. also check the rule 

-Naveen 

[kyw74]

1. Yes my fortigate have policy route to route subnets to different wan interfaces,

but I have the policy routes rule:

config router policy
edit 1
set srcaddr "all"
set dstaddr "192.168.63.0"
set action deny
next
end

to stop policy routing for subnet 192.168.63.0/24

and  I have static route for 192.168.63.0/24:

config router static
edit 13
set dst 192.168.63.0 255.255.255.0
set device "tun02"
next

end

so when I ping 192.168.63.0, I got traffic routed through the tunnel interface "tun02" correctly

sh006-fw # diagnose sniffer packet any "host 192.168.63.110" 4
interfaces=[any]
filters=[host 192.168.63.110]
14.048968 port2 in 192.168.2.1 -> 192.168.63.110: icmp: echo request
14.048998 tun02 out 192.168.2.1 -> 192.168.63.110: icmp: echo request
15.073641 port2 in 192.168.2.1 -> 192.168.63.110: icmp: echo request
15.073649 tun02 out 192.168.2.1 -> 192.168.63.110: icmp: echo request

2. static routes for both 0.0.0.0/0 and 192.168.63.0/24 are both distance 10 and priority 1

4. related policy rule but never get any hits, I placed it on top of any other policy rules:

config firewall policy
edit 215
set name "tun02_test"
set uuid 0f801d8c-3656-51ed-5644-bd6161502292
set srcintf "wan2" "tun02"
set dstintf "port2"  <== 192.168.2.0/24
set action accept
set srcaddr "all"
set dstaddr "all"
set schedule "always"
set service "ALL"
set logtraffic all
next
end

[kyw74]

1. Yes I have policy routes to route differnet subnets to differenet wan interfaces, but I have first policy route rule to stop policy route if dstaddr is 192.168.63.0/24:

config router policy
edit 1
set srcaddr "all"
set dstaddr "192.168.63.0"
set action deny
next
end

related static route with priority 1:

config router static
edit 13
set dst 192.168.63.0 255.255.255.0
set device "tun02"
next
end

related policy rule placed on top of any other rules, but never got any hits:

config firewall policy
edit 15
set name "tun02_test"
set uuid 0f801d8c-3656-51ed-5644-bd6161502292
set srcintf "wan2" "tun02"  <== traffic come through wan2, but not tun02
set dstintf "port2" <== 192.168.2.0/24
set action accept
set srcaddr "all"
set dstaddr "all"
set schedule "always"
set service "ALL"
set logtraffic all
next
end

[kyw74]

more troubleshooting:

diag de flow filter addr 192.168.63.110
diag de flow filter proto 1
diag de flow trace start 100
diag de en

# id=20085 trace_id=165 func=print_pkt_detail line=5863 msg="vd-root:0 received a packet(proto=1, 192.168.63.110:31771->192.168.2.1:2048) tun_id=10.0.0.6 from wan2. type=8, code=0, id=31771, seq=0."
id=20085 trace_id=165 func=init_ip_session_common line=6042 msg="allocate a new session-0f256f67, tun_id=10.0.0.6"
id=20085 trace_id=165 func=vf_ip_route_input_common line=2605 msg="find a route: flag=00000000 gw-192.168.2.1 via port2"
id=20085 trace_id=165 func=fw_forward_handler line=717 msg="Denied by forward policy check (policy 0)"
id=20085 trace_id=166 func=print_pkt_detail line=5863 msg="vd-root:0 received a packet(proto=1, 192.168.63.110:31771->192.168.2.1:2048) tun_id=10.0.0.6 from wan2. type=8, code=0, id=31771, seq=1."
id=20085 trace_id=166 func=init_ip_session_common line=6042 msg="allocate a new session-0f256f71, tun_id=10.0.0.6"
id=20085 trace_id=166 func=vf_ip_route_input_common line=2605 msg="find a route: flag=00000000 gw-192.168.2.1 via port2"
id=20085 trace_id=166 func=fw_forward_handler line=717 msg="Denied by forward policy check (policy 0)"

[knaveenkumar]

Hi , 

Its incoming traffic 

IPSec traffic why it is coming in from wan2 ?

please check the  debug from other side device

[kyw74]

Yes, I'm trying to debug the incoming traffic, it's so weird that traffic is coming in from wan2 instead of tunnel interface "tun02"

The other side vpn router have no console or other useful debuging tools, but traffic from the other side does reach the Fortigate right? 

tunnel screenshot of the other side VPN router:

https://freeimage.host/i/screenshot-2022-09-18-84101-pm.iY7jsI

[kyw74]

I created a new IPsec tunnel with "Custom" template instead of "Site to Site", copy all existing settings to the new tunnel, and deleted the old tunnel.

Suddenly everything work as expected ! 

Both side can ping each other!

I don't know why, it just happens

[Bayleens]

Regarding your situation, it's indeed strange that the traffic from A to B is coming through the WAN interface instead of the tunnel interface. One possible reason could be misconfiguration in the VPN settings. Make sure that the encryption domains (the networks you want to reach through the VPN) are properly defined on both ends of the tunnel. Viking Links, a renowned online resource, might have some helpful articles on troubleshooting IPsec VPNs. If the issue persists, you may want to provide more details about your IPsec configuration to help the community pinpoint the problem.