Question about weird policy route and connected route interactions on internal interfaces.

[Endiel]

Good morning!

So, I had recently posted a question regarding this... but have since figured it out-- though I don't fully understand how this is working and would like some clarification.  (This post is designed to help others who might have the same problem, but also in the hopes that somebody who's smarter than me can clarify how this is working.)

I've got a Fortigate 100D.  I run a high performance computing cluster, a company website, and now have our office LAN all sharing the same Fortigate.  To accommodate this, I have and pay for two separate internet circuits each with their own block of static IP's: one for the HPC cluster/website, and the other for the office.

(The following configuration uses 'fake' IP addresses for privacy reasons.)

The way I have it set up is in the classic dual WAN configuration.  

The 'wan2' interface, providing Internet to the cluster/website gets the first IP block: 163.213.3.147/255.255.255.240.

The 'port4' interface is being used as the second wan port, and gets the IP block: 163.213.5.50/255.255.255.240.

The 'office' interface is a VLAN interface configured on port4 that connects out to the office LAN, with the DHCP LAN of 10.100.100.0/255.255.255.0.

'wan2' and 'port4' are both set up the default static routes to 0.0.0.0/0.0.0.0 in the routing table with their listed GW's.  (This has been pruned to remove all routes that don't matter-- notice the connected routes on port4 and wan2 for our two IP blocks)

S*     0.0.0.0/0 [20/0] via 163.213.5.49, port4
                       [20/0] via 163.213.3.145, wan2

C       163.213.5.48/28 is directly connected, port4

C       163.213.3.144/28 is directly connected, wan2

Both default routes have the same priority/distance.  I then use policy routes to send all traffic from the 'Office' interface with an address of 10.100.100.0/255.255.255.0 out to 0.0.0.0/0.0.0.0 (the internet) through the wan interface on port4, while everything else goes out the main wan2.  That is working great.

But here is where things get funky.  Our webserver is located in the DMZ with a IP of 192.168.0.3 on port 8080.   and connects to our company website of 163.213.3.150:80 through a VIP on the 'wan2' interface that forwards the port 80 to port 8080 at the webserver.  Since 163.213.3.150 is a 'connected route', from what I can gather, the Fortigate will always want to try to forward directly through it internally.

So when I try to go to the office website from our LAN, it doesn't work.  Nothing gets through.  So I did a debug trace which revealed this funky routing behavior (10.100.100.70 being my laptop):

id=20085 trace_id=1611 func=print_pkt_detail line=4378 msg="vd-NAT received a packet(proto=6, 10.100.100.70:49244->163.213.3.150:80) from Office. flag (S), seq 1432385004, ack 0, win 65535"id=20085 trace_id=1611 func=init_ip_session_common line=4527 msg="allocate a new session-0167a417"
id=20085 trace_id=1611 func=fw_pre_route_handler line=174 msg="VIP-192.168.0.3:8080, outdev-unkown"
id=20085 trace_id=1611 func=__ip_session_run_tuple line=2537 msg="DNAT 163.213.3.150:80->192.168.0.3:8080"
id=20085 trace_id=1611 func=vf_ip4_route_input line=1586 msg="Match policy routing: to 163.213.5.49 via ifindex-10"
id=20085 trace_id=1611 func=vf_ip4_route_input line=1596 msg="find a route: flags=00000000 gw-163.213.5.49 via port4"
id=20085 trace_id=1611 func=fw_forward_handler line=545 msg="Denied by forward policy check (policy 0)"

As you can see, it's trying to go directly from the "Office" interface to wan2 without being squirted out the por4 interface first, but then is NAT'ing to the internal 192.168.0.3 address and trying to use the port4 gateway to get to it-- which gets denied.  Took me a while to figure out that was what the problem was... my initial inclination was that it should just forward out port4 given the policy route, but apparently the connected route takes precedence.

So I was tearing my hair out over this, because I had all the policies in the firewall setup correctly, etc.  So finally on a whim I added a policy route to point all 10.100.100.0/255.255.255.0 traffic coming from the "Office" interface to the DMZ interface for the block of 192.168.0.1/255.255.255.0 addresses.  And suddenly things began to work-- which didn't make sense until after I did another debug trace:

id=20085 trace_id=4696 func=print_pkt_detail line=4378 msg="vd-NAT received a packet(proto=6, 10.100.100.70:50535->163.213.3.150:80) from MKEI Office. flag , seq 1592356401, ack 0, win 65535"
id=20085 trace_id=4696 func=init_ip_session_common line=4527 msg="allocate a new session-016c7a7d"
id=20085 trace_id=4696 func=fw_pre_route_handler line=174 msg="VIP-192.168.0.3:8080, outdev-unkown"
id=20085 trace_id=4696 func=__ip_session_run_tuple line=2537 msg="DNAT 163.213.3.150:80->192.168.0.3:8080"
id=20085 trace_id=4696 func=vf_ip4_route_input line=1586 msg="Match policy routing: to 192.168.0.3 via ifindex-27"
id=20085 trace_id=4696 func=vf_ip4_route_input line=1596 msg="find a route: flags=00000000 gw-192.168.0.3 via dmz"
id=20085 trace_id=4696 func=fw_forward_handler line=670 msg="Allowed by Policy-8:"

See how it's looking for the route to the DMZ as that is the internal DNAT, even though it's routing through the wan2 interface.

So here's my question... I don't currently have a policy route that allows traffic from the "Office" interface to the dmz interface, which would necessitate the route... but since it's a connected route on the DMZ, why do I need to tell the policy route how to get there even though it apparently already prefers the connected route to wan2?  

Does this mean that this traffic is bouncing through "wan2" and then being directed to the DMZ interface, which is why it matches the policy?  Policy-8 is indeed the policy allowing traffic from "wan2" interface to the web server VIP.  I guess I'm just confused at the somewhat weird discretion of the fortigate to sometimes prefer policy routes and ignore certain connected routes, like from my "Office" interface  to the DMZ, but then to prefer other connected routes over my policy route, like to the "wan2" interface versus out to the internet on port4.

Anyways.  This is probably intended but still kind of weird-- I'm just glad I figured it out and hope it will help anybody also figure it out.

Thanks.