Hello!
Hoping someone might have some insight into this, because I'm stumped ...
Setup:
- 2 paths to the internet, via "port1" and down a VPN Tunnel.
- Default route is port1
- 3 Virtual Servers attached to port1
- Policy routes decide where to send outbound packets, port1 or tunnel.
All is working well until ...
If I change the default route to the tunnel instead of port1, the Virtual Servers stop working, even though I have policy routes that specifically force the outbound traffic for those virtual servers to port1, which is where the traffic comes in for them (always).
Everything else adapts pretty well ... just not the virtual servers.
When I look I see that traffic incoming to the virtual servers on port1 gets dropped with "reverse path check fail, drop". It doesn't find a path back ... even though there is a policy route?
Working incoming session:
Packet Trace #277 | 2023-09-07 11:21 | vd-root:0 received a packet(proto=6, 40.83.2.68:24801->10.1.9.7:443) tun_id=0.0.0.0 from port1. flag [S], seq 4182754383, ack 0, win 64240 |
Packet Trace #277 | 2023-09-07 11:21 | allocate a new session-011f89f8, tun_id=0.0.0.0 |
Packet Trace #277 | 2023-09-07 11:21 | in-[port1], out-[] |
Packet Trace #277 | 2023-09-07 11:21 | len=1 |
Packet Trace #277 | 2023-09-07 11:21 | checking gnum-100000 policy-3 |
Packet Trace #277 | 2023-09-07 11:21 | match vip-Public Web Site Virtual Server - HTTPS, naddr=10.1.13.7, nport=80 |
Packet Trace #277 | 2023-09-07 11:21 | matched policy-3, act=accept, vip=3, flag=100, sflag=2000400 |
Packet Trace #277 | 2023-09-07 11:21 | result: skb_flags-02000400, vid-3, ret-matched, act-accept, flag-00000100 |
Packet Trace #277 | 2023-09-07 11:21 | VIP-10.1.13.7:80, outdev-port1 |
Packet Trace #277 | 2023-09-07 11:21 | DNAT 10.1.9.7:443->10.1.13.7:80 |
Packet Trace #277 | 2023-09-07 11:21 | find a route: flag=00000000 gw-10.1.11.1 via port2 |
Packet Trace #277 | 2023-09-07 11:21 | in-[port1], out-[port2], skb_flags-020004c0, vid-3, app_id: 0, url_cat_id: 0 |
Packet Trace #277 | 2023-09-07 11:21 | gnum-100004, use int hash, slot=28, len=5 |
Packet Trace #277 | 2023-09-07 11:21 | checked gnum-100004 policy-42, ret-no-match, act-accept |
Packet Trace #277 | 2023-09-07 11:21 | checked gnum-100004 policy-48, ret-no-match, act-accept |
Packet Trace #277 | 2023-09-07 11:21 | checked gnum-100004 policy-48, ret-matched, act-accept |
Packet Trace #277 | 2023-09-07 11:21 | ret-matched |
Packet Trace #277 | 2023-09-07 11:21 | gnum-4e26, check-0000000080c870ab |
Packet Trace #277 | 2023-09-07 11:21 | checked gnum-4e26 policy-4294967295, ret-no-match, act-accept |
Packet Trace #277 | 2023-09-07 11:21 | checked gnum-4e26 policy-9, ret-matched, act-accept |
Packet Trace #277 | 2023-09-07 11:21 | policy-9 is matched, act-accept |
Packet Trace #277 | 2023-09-07 11:21 | gnum-4e26 check result: ret-matched, act-accept, flag-00a02008, flag2-00000000 |
Packet Trace #277 | 2023-09-07 11:21 | policy-48 is matched, act-accept |
Packet Trace #277 | 2023-09-07 11:21 | after iprope_captive_check(): is_captive-0, ret-matched, act-accept, idx-48 |
Packet Trace #277 | 2023-09-07 11:21 | after iprope_captive_check(): is_captive-0, ret-matched, act-accept, idx-48 |
Packet Trace #277 | 2023-09-07 11:21 | Allowed by Policy-48: AV |
Packet Trace #277 | 2023-09-07 11:21 | npu_state=0x1100, hook=4 |
Packet Trace #277 | 2023-09-07 11:21 | send to ips |
Packet Trace #277 | 2023-09-07 11:21 | send to application layer |
When I change the default route:
10:18:45
vd-root:0 received a packet(proto=6, 36.75.221.151:51442->10.1.9.7:443) tun_id=0.0.0.0 from port1. flag [S], seq 3117427467, ack 0, win 64240
10:18:45
allocate a new session-011e0cf6, tun_id=0.0.0.0
10:18:45
in-[port1], out-[]
10:18:45
len=1
10:18:45
checking gnum-100000 policy-3
10:18:45
match vip-Public Web Site Virtual Server - HTTPS, naddr=10.1.13.7, nport=80
10:18:45
matched policy-3, act=accept, vip=3, flag=100, sflag=2000400
10:18:45
result: skb_flags-02000400, vid-3, ret-matched, act-accept, flag-00000100
10:18:45
VIP-10.1.13.7:80, outdev-port1
10:18:45
DNAT 10.1.9.7:443->10.1.13.7:80
10:18:45
reverse path check fail, drop
10:18:45
trace
Policy route that should work I'm pretty sure ... :
config router policy
edit 13
set input-device "port2"
set src "10.1.13.7/255.255.255.255" "10.1.13.5/255.255.255.255" "10.1.13.6/255.255.255.255" "10.1.13.8/255.255.255.255"
set dstaddr "Internet"
set gateway 10.1.9.1
set output-device "port1"
next
end
10.1.9.7 is the Virtual Server IP, 10.1.13.7 is the backend server IP.
Are policy routes not used to check return path??
Thanks!
Nominating a forum post submits a request to create a new Knowledge Article based on the forum post topic. Please ensure your nomination includes a solution within the reply.
Hi @jdoyon ,
The answer to your problem is
"If no routes are found in the routing table, then the policy route does not match the packet. The FortiGate continues down the policy route list until it reaches the end. If no matches are found, then the FortiGate does a route lookup using the routing table."
So you need a route in routing table for Policy route to work, but this route need not be the best route. So you can have two default route and the one using tunnel with Higher Preference (Low Priority Value and same distance) and the one using port1 with Lower preference (high Priority and same distance as other default route).
Best Regards,
San
Bu there are static routes, 2 of relevance here, they are:
1 static default route, goes out port1, priority 10.
1 static default route, goes down the tunnel, priority 1.
When the tunnel route is disabled, all is well. When I enable the tunnel route, the above mentioned problem occurs, and suddenly there's no return path ... even though there's a policy route that says there should be (Said policy route has data flow through it when using the port1 default route, so that's a good sign it "works"?).
Hi @jdoyon
As per the logs traffic is failing due to reverse path check fail, which means when the Firewall check for a route back to the source 40.83.2.68 via Port1, it doesn't find a route in the Kernel routing table and policy route will not be considered here as it only look at the routing table and moreover your Policy criteria say input interface is Port2 where here traffic in entering on Port1. Policy route will be effective in the outbound traffic scenarios based on the criteria you defined.
If you don't mind could you please provide below information with and without tunnel disabled, because I strongly doubt there is no route in the Routing table for destination 40.83.2.68 when tunnel is enabled.
get router info routing-table all
get router info kernel
Best Regards,
Ah OK I think I understand the problem then ... I'll double check, but if you're right, then how should I handle my scenario?
- I want default outbound internet traffic to go down the tunnel,
- But I want exceptions to go out port1 (This scenario I've had working in the past).
- And I want any traffic that came into port1, to go back out port1 ... preventing asymmetric routing that could occur because traffic came into port1, but could return down the tunnel. (This is what I'm trying to do now).
I can't put a static route for the port1 traffic, because the destination is the internet, and the traffic intended to go out port1 is ONLY traffic that came IN from port1, but in the routes I distinguish that traffic because I know where it went internally ... so I say traffic that's coming back into port2, from these hosts, I know must go to port1. The internet traffic coming into port1 is handled by virtual servers.
Thoughts?
Hi @jdoyon ,
It should be doable, as I mentioned, you need two default routes.
Ex:
Route1:
0.0.0.0/0 next-hop Tunnel - Distance 0, Priority 0
Route 2:
0.0.0.0/0 next-hop Port1- Distance 0, Priority 10
This will make sure both the routes present in the routing table and your RPF check is successful. Please also review below article about how FortiGate perform RPF checks . Strict check option enabled or disabled will change the behavior in which it handles RPF check.
As per document - "The default feasible RPF mode checks only for the existence of at least one active route back to the source using the incoming interface. The strict RPF check ensures the best route back to the source is used as the incoming interface." - So you need to review your configuration and see if the firewall is set to Strict Check or not.
https://docs.fortinet.com/document/fortigate/6.2.15/cookbook/139692/routing-concepts#Reverse
Please let me know if you have any additional questions.
Best Regards,
That's what I've been doing ... I HAVE those two routes, configured that way but the RPF check fails as detailed in the OP.
If I do as you say, which is what I've been trying, then traffic coming into port1 does not find a return path, because the main route back will be down the tunnel, and not out port1, and it will never move on to check policy routes.
Packet comes in to port1, FGT looks for the return path, the return path is the tunnel, so I fail. There is another *possible* route back, of lesser priority (bigger number), but it never sees that one I guess ... though it sounds like it should?
I have confirmed that Strict RPF is NOT enabled:
OlCcFWLCore-FGT-A (settings) # show full-configuration
config system settings
...
set strict-src-check disable
...
Here is some of the routing details with both routes on as you described:
OlCcFWLCore-FGT-A # get router info routing-table all
Routing table for VRF=0
S* 0.0.0.0/0 [10/0] via ocol-paz-fw-2 tunnel <obfuscated>, [1/0]
[10/0] is a summary, Null, [1/0]
[10/0] via 10.1.9.1, port1, [10/0]
You see both routes, with the different priorities are present.
And in the kernel:
tab=254 vf=0 scope=0 type=1 proto=11 prio=1 0.0.0.0/0.0.0.0/0->0.0.0.0/0 pref=0.0.0.0 gwy=<obfuscated> dev=13(ocol-paz-fw-2)
tab=254 vf=0 scope=0 type=6 proto=11 prio=1 0.0.0.0/0.0.0.0/0->0.0.0.0/0 pref=0.0.0.0 gwy=0.0.0.0 dev=0
tab=254 vf=0 scope=0 type=1 proto=11 prio=10 0.0.0.0/0.0.0.0/0->0.0.0.0/0 pref=0.0.0.0 gwy=10.1.9.1 dev=3(port1)
Hi @jdoyon
Thank you for sharing the routing information and I could see your configuration same as per the provided documents I have shared earlier. If you don't mind may I know the Fort iOS version you run in your setup. Maybe I can quickly test this one and share you the results
Best Regards
Version is v7.2.5 build1517 (Feature) ... It is an Azure VM.
Created on 09-11-2023 10:23 AM Edited on 09-11-2023 10:25 AM
Hi
Indeed, as everyone said, you must maintain the default route for both links in the routing table for the traffic to reach the Virtual Server.
You can refer to this article for a clear picture.
if you suspect reply traffic is honoring the policy route. Please verify whether you have Asymmetric Routing or Auxiliary Sessions enabled. When enabled, a routing lookup performed for reply traffic will choose the policy route.
Hope that helps,
Kind Regards,
Bijay Prakash Ghising
Select Forum Responses to become Knowledge Articles!
Select the “Nominate to Knowledge Base” button to recommend a forum post to become a knowledge article.
User | Count |
---|---|
1710 | |
1093 | |
752 | |
447 | |
231 |
The Fortinet Security Fabric brings together the concepts of convergence and consolidation to provide comprehensive cybersecurity protection for all users, devices, and applications and across all network edges.
Copyright 2024 Fortinet, Inc. All Rights Reserved.