Problem with Virtual Server and Policy Route: reverse path check fail, drop

[jdoyon]

Hello!

Hoping someone might have some insight into this, because I'm stumped ...

Setup:

- 2 paths to the internet, via "port1" and down a VPN Tunnel.

- Default route is port1

- 3 Virtual Servers attached to port1

- Policy routes decide where to send outbound packets, port1 or tunnel.

All is working well until ...

If I change the default route to the tunnel instead of port1, the Virtual Servers stop working, even though I have policy routes that specifically force the outbound traffic for those virtual servers to port1, which is where the traffic comes in for them (always).

Everything else adapts pretty well ... just not the virtual servers.

When I look I see that traffic incoming to the virtual servers on port1 gets dropped with "reverse path check fail, drop". It doesn't find a path back ... even though there is a policy route?

Working incoming session:

Packet Trace #277	2023-09-07 11:21	vd-root:0 received a packet(proto=6, 40.83.2.68:24801->10.1.9.7:443) tun_id=0.0.0.0 from port1. flag [S], seq 4182754383, ack 0, win 64240
Packet Trace #277	2023-09-07 11:21	allocate a new session-011f89f8, tun_id=0.0.0.0
Packet Trace #277	2023-09-07 11:21	in-[port1], out-[]
Packet Trace #277	2023-09-07 11:21	len=1
Packet Trace #277	2023-09-07 11:21	checking gnum-100000 policy-3
Packet Trace #277	2023-09-07 11:21	match vip-Public Web Site Virtual Server - HTTPS, naddr=10.1.13.7, nport=80
Packet Trace #277	2023-09-07 11:21	matched policy-3, act=accept, vip=3, flag=100, sflag=2000400
Packet Trace #277	2023-09-07 11:21	result: skb_flags-02000400, vid-3, ret-matched, act-accept, flag-00000100
Packet Trace #277	2023-09-07 11:21	VIP-10.1.13.7:80, outdev-port1
Packet Trace #277	2023-09-07 11:21	DNAT 10.1.9.7:443->10.1.13.7:80
Packet Trace #277	2023-09-07 11:21	find a route: flag=00000000 gw-10.1.11.1 via port2
Packet Trace #277	2023-09-07 11:21	in-[port1], out-[port2], skb_flags-020004c0, vid-3, app_id: 0, url_cat_id: 0
Packet Trace #277	2023-09-07 11:21	gnum-100004, use int hash, slot=28, len=5
Packet Trace #277	2023-09-07 11:21	checked gnum-100004 policy-42, ret-no-match, act-accept
Packet Trace #277	2023-09-07 11:21	checked gnum-100004 policy-48, ret-no-match, act-accept
Packet Trace #277	2023-09-07 11:21	checked gnum-100004 policy-48, ret-matched, act-accept
Packet Trace #277	2023-09-07 11:21	ret-matched
Packet Trace #277	2023-09-07 11:21	gnum-4e26, check-0000000080c870ab
Packet Trace #277	2023-09-07 11:21	checked gnum-4e26 policy-4294967295, ret-no-match, act-accept
Packet Trace #277	2023-09-07 11:21	checked gnum-4e26 policy-9, ret-matched, act-accept
Packet Trace #277	2023-09-07 11:21	policy-9 is matched, act-accept
Packet Trace #277	2023-09-07 11:21	gnum-4e26 check result: ret-matched, act-accept, flag-00a02008, flag2-00000000
Packet Trace #277	2023-09-07 11:21	policy-48 is matched, act-accept
Packet Trace #277	2023-09-07 11:21	after iprope_captive_check(): is_captive-0, ret-matched, act-accept, idx-48
Packet Trace #277	2023-09-07 11:21	after iprope_captive_check(): is_captive-0, ret-matched, act-accept, idx-48
Packet Trace #277	2023-09-07 11:21	Allowed by Policy-48: AV
Packet Trace #277	2023-09-07 11:21	npu_state=0x1100, hook=4
Packet Trace #277	2023-09-07 11:21	send to ips
Packet Trace #277	2023-09-07 11:21	send to application layer

When I change the default route:

10:18:45
vd-root:0 received a packet(proto=6, 36.75.221.151:51442->10.1.9.7:443) tun_id=0.0.0.0 from port1. flag [S], seq 3117427467, ack 0, win 64240
10:18:45
allocate a new session-011e0cf6, tun_id=0.0.0.0
10:18:45
in-[port1], out-[]
10:18:45
len=1
10:18:45
checking gnum-100000 policy-3
10:18:45
match vip-Public Web Site Virtual Server - HTTPS, naddr=10.1.13.7, nport=80
10:18:45
matched policy-3, act=accept, vip=3, flag=100, sflag=2000400
10:18:45
result: skb_flags-02000400, vid-3, ret-matched, act-accept, flag-00000100
10:18:45
VIP-10.1.13.7:80, outdev-port1
10:18:45
DNAT 10.1.9.7:443->10.1.13.7:80
10:18:45
reverse path check fail, drop
10:18:45
trace

Policy route that should work I'm pretty sure ... :

config router policy
edit 13
set input-device "port2"
set src "10.1.13.7/255.255.255.255" "10.1.13.5/255.255.255.255" "10.1.13.6/255.255.255.255" "10.1.13.8/255.255.255.255"
set dstaddr "Internet"
set gateway 10.1.9.1
set output-device "port1"
next
end

10.1.9.7 is the Virtual Server IP, 10.1.13.7 is the backend server IP.

Are policy routes not used to check return path??

Thanks!

[jdoyon]

I am doing exactly as the first link says. 2 routes, same AD, different priority.

See the routing CLI commands and outputs posted above.

[akristof]

Hi.

Maybe I've missed something. But if you have default route via port1, everything is working as expected. But if you remove default route (port1) and keeps default via VPN and you add policy route (input port2, output port1, ...) then it is not working? You expect that the policy route will be used for RPF check?

[jdoyon]

It is not clear form the docs whether policy routes are looked at for RPF from the docs IIRC (but i could be wrong). From what was said above, static / kernel routes have to match first on RPF check? In which case, the port1 route, which is still there, but at lower priority than the tunnel, should still be found when RPF is not "strict", but that does not seem to happen. If at least that worked, the RPF would "pass", and when the traffic ACTUALLY returns, the policy route would apply ... ?

Funny side note: I could've sworn I was at 7.4.1 ... now it says 7.2.5 ... and apparently there was a VM "crash' or something last night ... hummm ...

[gsekar]

Kindly refer to the below article

https://docs.fortinet.com/document/fortigate/6.4.3/administration-guide/144044/policy-routes#:~:text....