Routing Internet Traffic through non-WAN port

[rjclay]

Cisco to Fortigate convert here and running into a problem.  I'm trying to simulate something I can do on my Cisco devices by changing a single line of code (these are not firewall/router combo devices, the Fortigates are) and see if you can help me do the same with Fortigate.

Cisco router has two default routes with different costs, lowest cost one points to local ISP and the other back to corporate over a MPLS with service provider.  I separate 'corporate' traffic from internet traffic with a static route 10.2.0.0 pointing to MPLS (all corp traffic starts with 10.2.x.x) and default route points to ISP.  This works great and if internet goes down at location I can reroute internet traffic back through corporate by increasing the cost of default route going to ISP over that of the one leading to MPLS/Corporate.  This works and allows me to use that MPLS/Corporate connection to route internet traffic to branch until their ISP comes back up.

I tried doing this on Fortigate but am running into a problem.  Internet is connected to WAN port, NAT enabled and works great.  The MPLS/Corporate plugged into port 2 but NAT has to be turned off for it to communicate. Lan in port 1.  I created a static route 10.2.x.x pointing to MPLS/Corporate and a 0.0.0.0 pointing to WAN and everything works. Problem is when/if local ISP goes down and I change the 0.0.0.0 default route to point to MPLS/Corporate Port 2, it can't route the internet traffic.

I'm assuming I can get this concept to work but can't use the same logic as the stand alone Cisco router due to the NAT not being enabled on the MPLS/Corporate port, does anyone have an idea how I could temp route internet traffic to/from a branch location over that non-nat connection back to MPLS/Corporate? Thanks for the time!

[Toshi_Esumi]

It should work. First make sure you have a proper policy for the internet traffic going to MPLS interface, not only for traffic destined to 10.2.x.x.

Then make sure you enable SNAT reroute under global config.

config system global

  set snat-route-change enable

end

This is only for existing/ongoing sessions. And if it still doesn't work while your routing table looks correct (I'm assuming you change the distance for the second default route) in "get router info routing-t all", you probably need to run "flow debugging" to see why packets don't go the MPLS interface.

diag debug reset

diag debug flow filter clear

diag debug enable    <-- need this if not at console

diag debug flow filter addr <IP_address_for_ping_destination>

diag debug flow trace start 10

Toshi

[rjclay]

Thanks for the info, I'll give it a shot at my test site across town soon!  In the meantime can you tell me more about the "have a proper policy for the internet traffic going to MPLS interface" you mentioned? I think I have that under consideration but I'm also convinced there is some small obvious thing I'm missing so any additional info will be consumed ;D

[Toshi_Esumi]

FGT's FW policies work with source and destination interface pairs. You must have an outgoing policy from a LAN side interface to the WAN interface for "all" destinations with a NAT, which is working. You need to have the same policy from the LAN side interface to the MPLS interface for "all" destinations without a NAT. 

I'm assuming you already have a policy from the LAN to MPLS for 10.2.x.x destination. So you could just remove the destination address restriction of the existing policy instead of creating another policy.

That's what I meant.

Toshi

[rjclay]

Was going to DM you this but figured it might help someone else looking for similar!  I'm not ready to change anything just yet, want to be on site in case I break something, but here is what shows with sh firewall policy on that unit (Lumen=MPLS) :

WH2-RTR # sh firewall policy
config firewall policy
edit 3
set name "Lumen to Lan"
set uuid xxx
set srcintf "lan3"
set dstintf "lan"
set srcaddr "all"
set dstaddr "all"
set action accept
set schedule "always"
set service "ALL"
set utm-status enable
set ssl-ssh-profile "certificate-inspection"
set av-profile "default"
set comments " (Copy of Lan to Lumen)"
next
edit 1
set uuid xxx
set srcintf "lan"
set dstintf "wan"
set srcaddr "all"
set dstaddr "all"
set action accept
set schedule "always"
set service "ALL"
set nat enable
next
edit 2
set name "Lan to Lumen"
set uuid xxx
set srcintf "lan"
set dstintf "lan3"
set srcaddr "all"
set dstaddr "all"
set action accept
set schedule "always"
set service "ALL"
set utm-status enable
set ssl-ssh-profile "certificate-inspection"
set av-profile "default"
next
end

[Toshi_Esumi]

Looks fine to me but is your FGT a smaller model like 40F? Then, you removed "lan3" from "lan" hard-switch interface to be independent?

Toshi

[rjclay]

Oh yes sorry it is a 40F and Lumen is plugged into port 3 instead of 2, just wanted that bit of space to visualize the LAN from MPLS hah. our LAN is coming out of port 1

[Toshi_Esumi]

You can make the 'a' port as your MPLS port if you want. Just need to remove "fortilink" related config. That's how I have with my home 40F's 'a' port as secondary WAN port.

Toshi

[rjclay]

Hey sorry for delay but finally got over to do some testing.  Oddly enough changing the cost of the second default gateway to lan3 didn't cause the traffic to switch over, had to take the WAN interface down to make it try and use the second default route but that's not so terrible.  With the WAN down and a default route pointing to Lumen/lan3 I ran the diag and got this result, it still didn't work, this is also after setting the global snat setting you mentioned, thanks!:

WH2-RTR # diag debug enable

WH2-RTR # diag debug flow filter addr 8.8.8.8

WH2-RTR # diag debug flow trace start 20

WH2-RTR # id=20085 trace_id=41 func=print_pkt_detail line=5742 msg="vd-root:0 received a packet(proto=17, 10.2.18.57:61113->8.8.8.8:53) from lan. "
id=20085 trace_id=41 func=resolve_ip_tuple_fast line=5823 msg="Find an existing session, id-00e42b2a, original direction"
id=20085 trace_id=41 func=npu_handle_session44 line=1165 msg="Trying to offloading session from lan to lan3, skb.npu_flag=00000400 ses.state=00032200 ses.npu_state=0x00103094"
id=20085 trace_id=41 func=fw_forward_dirty_handler line=395 msg="state=00032200, state2=00000000, npu_state=00103094"
id=20085 trace_id=41 func=__ip_session_run_tuple line=3537 msg="run helper-dns-udp(dir=original)"
id=20085 trace_id=41 func=ipd_post_route_handler line=490 msg="out lan3 vwl_zone_id 0, state2 0x0, quality 0.
"
id=20085 trace_id=42 func=print_pkt_detail line=5742 msg="vd-root:0 received a packet(proto=17, 10.2.18.57:56806->8.8.8.8:53) from lan. "
id=20085 trace_id=42 func=init_ip_session_common line=5913 msg="allocate a new session-00e42b5f"
id=20085 trace_id=42 func=vf_ip_route_input_common line=2621 msg="find a route: flag=04000000 gw-8.8.8.8 via lan3"
id=20085 trace_id=42 func=fw_forward_handler line=799 msg="Allowed by Policy-2:"
id=20085 trace_id=42 func=__ip_session_run_tuple line=3537 msg="run helper-dns-udp(dir=original)"
id=20085 trace_id=42 func=ipd_post_route_handler line=490 msg="out lan3 vwl_zone_id 0, state2 0x0, quality 0.
"
id=20085 trace_id=43 func=print_pkt_detail line=5742 msg="vd-root:0 received a packet(proto=17, 10.2.18.57:65490->8.8.8.8:53) from lan. "
id=20085 trace_id=43 func=resolve_ip_tuple_fast line=5823 msg="Find an existing session, id-00e42b2d, original direction"
id=20085 trace_id=43 func=npu_handle_session44 line=1165 msg="Trying to offloading session from lan to lan3, skb.npu_flag=00000400 ses.state=00032200 ses.npu_state=0x00103094"
id=20085 trace_id=43 func=fw_forward_dirty_handler line=395 msg="state=00032200, state2=00000000, npu_state=00103094"
id=20085 trace_id=43 func=__ip_session_run_tuple line=3537 msg="run helper-dns-udp(dir=original)"
id=20085 trace_id=43 func=ipd_post_route_handler line=490 msg="out lan3 vwl_zone_id 0, state2 0x0, quality 0.
"
id=20085 trace_id=44 func=print_pkt_detail line=5742 msg="vd-root:0 received a packet(proto=17, 10.2.18.57:56806->8.8.8.8:53) from lan. "
id=20085 trace_id=44 func=resolve_ip_tuple_fast line=5823 msg="Find an existing session, id-00e42b5f, original direction"
id=20085 trace_id=44 func=npu_handle_session44 line=1165 msg="Trying to offloading session from lan to lan3, skb.npu_flag=00000400 ses.state=00012200 ses.npu_state=0x00003094"
id=20085 trace_id=44 func=fw_forward_dirty_handler line=395 msg="state=00032200, state2=00000000, npu_state=00103094"
id=20085 trace_id=44 func=__ip_session_run_tuple line=3537 msg="run helper-dns-udp(dir=original)"
id=20085 trace_id=44 func=ipd_post_route_handler line=490 msg="out lan3 vwl_zone_id 0, state2 0x0, quality 0.
"
id=20085 trace_id=45 func=print_pkt_detail line=5742 msg="vd-root:0 received a packet(proto=17, 10.2.18.57:56806->8.8.8.8:53) from lan. "
id=20085 trace_id=45 func=resolve_ip_tuple_fast line=5823 msg="Find an existing session, id-00e42b5f, original direction"
id=20085 trace_id=45 func=npu_handle_session44 line=1165 msg="Trying to offloading session from lan to lan3, skb.npu_flag=00000400 ses.state=00032200 ses.npu_state=0x00103094"
id=20085 trace_id=45 func=fw_forward_dirty_handler line=395 msg="state=00032200, state2=00000000, npu_state=00103094"