Problem with Virtual Server and Policy Route: reverse path check fail, drop

[jdoyon]

Hello!

Hoping someone might have some insight into this, because I'm stumped ...

Setup:

- 2 paths to the internet, via "port1" and down a VPN Tunnel.

- Default route is port1

- 3 Virtual Servers attached to port1

- Policy routes decide where to send outbound packets, port1 or tunnel.

All is working well until ...

If I change the default route to the tunnel instead of port1, the Virtual Servers stop working, even though I have policy routes that specifically force the outbound traffic for those virtual servers to port1, which is where the traffic comes in for them (always).

Everything else adapts pretty well ... just not the virtual servers.

When I look I see that traffic incoming to the virtual servers on port1 gets dropped with "reverse path check fail, drop". It doesn't find a path back ... even though there is a policy route?

Working incoming session:

Packet Trace #277	2023-09-07 11:21	vd-root:0 received a packet(proto=6, 40.83.2.68:24801->10.1.9.7:443) tun_id=0.0.0.0 from port1. flag [S], seq 4182754383, ack 0, win 64240
Packet Trace #277	2023-09-07 11:21	allocate a new session-011f89f8, tun_id=0.0.0.0
Packet Trace #277	2023-09-07 11:21	in-[port1], out-[]
Packet Trace #277	2023-09-07 11:21	len=1
Packet Trace #277	2023-09-07 11:21	checking gnum-100000 policy-3
Packet Trace #277	2023-09-07 11:21	match vip-Public Web Site Virtual Server - HTTPS, naddr=10.1.13.7, nport=80
Packet Trace #277	2023-09-07 11:21	matched policy-3, act=accept, vip=3, flag=100, sflag=2000400
Packet Trace #277	2023-09-07 11:21	result: skb_flags-02000400, vid-3, ret-matched, act-accept, flag-00000100
Packet Trace #277	2023-09-07 11:21	VIP-10.1.13.7:80, outdev-port1
Packet Trace #277	2023-09-07 11:21	DNAT 10.1.9.7:443->10.1.13.7:80
Packet Trace #277	2023-09-07 11:21	find a route: flag=00000000 gw-10.1.11.1 via port2
Packet Trace #277	2023-09-07 11:21	in-[port1], out-[port2], skb_flags-020004c0, vid-3, app_id: 0, url_cat_id: 0
Packet Trace #277	2023-09-07 11:21	gnum-100004, use int hash, slot=28, len=5
Packet Trace #277	2023-09-07 11:21	checked gnum-100004 policy-42, ret-no-match, act-accept
Packet Trace #277	2023-09-07 11:21	checked gnum-100004 policy-48, ret-no-match, act-accept
Packet Trace #277	2023-09-07 11:21	checked gnum-100004 policy-48, ret-matched, act-accept
Packet Trace #277	2023-09-07 11:21	ret-matched
Packet Trace #277	2023-09-07 11:21	gnum-4e26, check-0000000080c870ab
Packet Trace #277	2023-09-07 11:21	checked gnum-4e26 policy-4294967295, ret-no-match, act-accept
Packet Trace #277	2023-09-07 11:21	checked gnum-4e26 policy-9, ret-matched, act-accept
Packet Trace #277	2023-09-07 11:21	policy-9 is matched, act-accept
Packet Trace #277	2023-09-07 11:21	gnum-4e26 check result: ret-matched, act-accept, flag-00a02008, flag2-00000000
Packet Trace #277	2023-09-07 11:21	policy-48 is matched, act-accept
Packet Trace #277	2023-09-07 11:21	after iprope_captive_check(): is_captive-0, ret-matched, act-accept, idx-48
Packet Trace #277	2023-09-07 11:21	after iprope_captive_check(): is_captive-0, ret-matched, act-accept, idx-48
Packet Trace #277	2023-09-07 11:21	Allowed by Policy-48: AV
Packet Trace #277	2023-09-07 11:21	npu_state=0x1100, hook=4
Packet Trace #277	2023-09-07 11:21	send to ips
Packet Trace #277	2023-09-07 11:21	send to application layer

When I change the default route:

10:18:45
vd-root:0 received a packet(proto=6, 36.75.221.151:51442->10.1.9.7:443) tun_id=0.0.0.0 from port1. flag [S], seq 3117427467, ack 0, win 64240
10:18:45
allocate a new session-011e0cf6, tun_id=0.0.0.0
10:18:45
in-[port1], out-[]
10:18:45
len=1
10:18:45
checking gnum-100000 policy-3
10:18:45
match vip-Public Web Site Virtual Server - HTTPS, naddr=10.1.13.7, nport=80
10:18:45
matched policy-3, act=accept, vip=3, flag=100, sflag=2000400
10:18:45
result: skb_flags-02000400, vid-3, ret-matched, act-accept, flag-00000100
10:18:45
VIP-10.1.13.7:80, outdev-port1
10:18:45
DNAT 10.1.9.7:443->10.1.13.7:80
10:18:45
reverse path check fail, drop
10:18:45
trace

Policy route that should work I'm pretty sure ... :

config router policy
edit 13
set input-device "port2"
set src "10.1.13.7/255.255.255.255" "10.1.13.5/255.255.255.255" "10.1.13.6/255.255.255.255" "10.1.13.8/255.255.255.255"
set dstaddr "Internet"
set gateway 10.1.9.1
set output-device "port1"
next
end

10.1.9.7 is the Virtual Server IP, 10.1.13.7 is the backend server IP.

Are policy routes not used to check return path??

Thanks!

[saneeshpv_FTNT]

Hi @jdoyon ,

The answer to your problem is

"If no routes are found in the routing table, then the policy route does not match the packet. The FortiGate continues down the policy route list until it reaches the end. If no matches are found, then the FortiGate does a route lookup using the routing table."

So you need a route in routing table for Policy route to work, but this route need not be the best route. So you can have two default route and the one using tunnel with Higher Preference (Low Priority Value and same distance) and the one using port1 with Lower preference (high Priority and same distance as other default route).

https://docs.fortinet.com/document/fortigate/7.4.1/administration-guide/144044/policy-routes#:~:text....

Best Regards,

San

[jdoyon]

Bu there are static routes, 2 of relevance here, they are:

1 static default route, goes out port1, priority 10.

1 static default route, goes down the tunnel, priority 1.

When the tunnel route is disabled, all is well. When I enable the tunnel route, the above mentioned problem occurs, and suddenly there's no return path ... even though there's a policy route that says there should be (Said policy route has data flow through it when using the port1 default route, so that's a good sign it "works"?).

[saneeshpv_FTNT]

Hi @jdoyon 

As per the logs traffic is failing due to reverse path check fail, which means when the Firewall check for a route back to the source 40.83.2.68 via Port1, it doesn't find a route in the Kernel routing table and policy route will not be considered here as it only look at the routing table and moreover your Policy criteria say input interface is Port2 where here traffic in entering on Port1. Policy route will be effective in the outbound traffic scenarios based on the criteria you defined.  

If you don't mind could you please provide below information with and without tunnel disabled, because I strongly doubt there is no route in the Routing table for destination 40.83.2.68 when tunnel is enabled. 

get router info routing-table all

get router info kernel

Best Regards,

[jdoyon]

Ah OK I think I understand the problem then ... I'll double check, but if you're right, then how should I handle my scenario?

- I want default outbound internet traffic to go down the tunnel,

- But I want exceptions to go out port1 (This scenario I've had working in the past).

- And I want any traffic that came into port1, to go back out port1 ... preventing asymmetric routing that could occur because traffic came into port1, but could return down the tunnel. (This is what I'm trying to do now).

I can't put a static route for the port1 traffic, because the destination is the internet, and the traffic intended to go out port1 is ONLY traffic that came IN from port1, but in the routes I distinguish that traffic because I know where it went internally ... so I say traffic that's coming back into port2, from these hosts, I know must go to port1. The internet traffic coming into port1 is handled by virtual servers.

Thoughts?

[saneeshpv_FTNT]

Hi @jdoyon ,

It should be doable, as I mentioned, you need two default routes.

Ex:

Route1:

0.0.0.0/0 next-hop Tunnel - Distance 0, Priority 0

Route 2:

0.0.0.0/0 next-hop Port1- Distance 0, Priority 10

https://community.fortinet.com/t5/FortiGate/Technical-Note-Setting-priority-on-static-default-routes...

This will make sure both the routes present in the routing table and your RPF check is successful. Please also review below article about how FortiGate perform RPF checks . Strict check option enabled or disabled will change the behavior in which it handles RPF check.

As per document - "The default feasible RPF mode checks only for the existence of at least one active route back to the source using the incoming interface. The strict RPF check ensures the best route back to the source is used as the incoming interface." - So you need to review your configuration and see if the firewall is set to Strict Check or not.

https://docs.fortinet.com/document/fortigate/6.2.15/cookbook/139692/routing-concepts#Reverse

Please let me know if you have any additional questions.

Best Regards,

[jdoyon]

That's what I've been doing ... I HAVE those two routes, configured that way but the RPF check fails as detailed in the OP.

If I do as you say, which is what I've been trying, then traffic coming into port1 does not find a return path, because the main route back will be down the tunnel, and not out port1, and it will never move on to check policy routes.

Packet comes in to port1, FGT looks for the return path, the return path is the tunnel, so I fail. There is another *possible* route back, of lesser priority (bigger number), but it never sees that one I guess ... though it sounds like it should?

I have confirmed that Strict RPF is NOT enabled:

OlCcFWLCore-FGT-A (settings) # show full-configuration
config system settings
...
set strict-src-check disable

...

Here is some of the routing details with both routes on as you described:

OlCcFWLCore-FGT-A # get router info routing-table all

Routing table for VRF=0
S* 0.0.0.0/0 [10/0] via ocol-paz-fw-2 tunnel <obfuscated>, [1/0]
[10/0] is a summary, Null, [1/0]
[10/0] via 10.1.9.1, port1, [10/0]

You see both routes, with the different priorities are present.

And in the kernel:

tab=254 vf=0 scope=0 type=1 proto=11 prio=1 0.0.0.0/0.0.0.0/0->0.0.0.0/0 pref=0.0.0.0 gwy=<obfuscated> dev=13(ocol-paz-fw-2)
tab=254 vf=0 scope=0 type=6 proto=11 prio=1 0.0.0.0/0.0.0.0/0->0.0.0.0/0 pref=0.0.0.0 gwy=0.0.0.0 dev=0
tab=254 vf=0 scope=0 type=1 proto=11 prio=10 0.0.0.0/0.0.0.0/0->0.0.0.0/0 pref=0.0.0.0 gwy=10.1.9.1 dev=3(port1)

[saneeshpv_FTNT]

Hi @jdoyon 

Thank you for sharing the routing information and I could see your configuration same as per the provided documents I have shared earlier. If you don't mind may I know the Fort iOS version you run in your setup. Maybe I can quickly test this one and share you the results

Best Regards

[jdoyon]

Version is v7.2.5 build1517 (Feature) ... It is an Azure VM.

[Bjay_Prakash_Ghising]

Hi

Indeed, as everyone said, you must maintain the default route for both links in the routing table for the traffic to reach the Virtual Server. 

You can refer to this article for a clear picture.

https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-VIP-not-working-when-configured-on-s...

https://community.fortinet.com/t5/FortiGate/Technical-Note-Routing-behavior-depending-on-distance-an...

if you suspect reply traffic is honoring the policy route. Please verify whether you have Asymmetric Routing or Auxiliary Sessions enabled. When enabled, a routing lookup performed for reply traffic will choose the policy route.

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Policy-Routing-Enhancements-for-Traffic-in...

Hope that helps, 

Kind Regards, 

Bijay Prakash Ghising