VIP doesn't work with my Second ISP WAN Link

[mahmoudmaaiteh]

Hello everybody,

I have been struggling to get this done for about three days with no result,

i have Fortigate 501E, Firmware v5.6.0.

i have 2 ISP connections each one with different public IP ranges, ISP A with public subnet X.X.X.X/X is connected directly from fortigate port 1 to the ISP MPLS, the fortigate interface port 1 configuration is straight through, physical interface with static IP address X.X.X.2/X and its role is WAN. for this interface everything is working fine and virtual IPs from the same subnet (X.X.X.X/X) are reachable via this interface.

my second WAN link Y.Y.Y.Y/Y is somehow complicated, FortiGate is connected to an Outside Switch, the fortigate interface port 8 which is connected to the outside switch has two VLAN interfaces configured under port 8, the physical port doesn't have any configuration nor IP Address, the two vlan interfaces configuration is listed bellow:

    edit "Main Link"

        set vdom "root"

        set ip 10.100.100.2 255.255.255.248

        set allowaccess ping

        set role wan

        set snmp-index 29

        set interface "port8"

        set vlanid 1000

    next

edit "Backup Link"

        set vdom "root"

        set ip 10.100.200.2 255.255.255.248

        set allowaccess ping

        set role wan

        set snmp-index 30

        set interface "port8"

        set vlanid 1001

    next

the Outside switch interface (connected to FortiGate) is configured as trunk with both vlan 1000,1001 allowed. 

Also the switch has two interfaces each is connected to a different MPLS ( both MPLSs are owned by the same ISP) one interface is access vlan 1000 with the opposite interface on the MPLS has an IP Address of 10.100.100.1, and the other interface is access vlan 1001 with MPLS interface 10.100.200.1. those links are for the same IP Range and the same WAN Link ( active/ standby) with SLA configured from ISP side on both of the links and a link monitor from fortigate side. 

im planning to use the second link Y.Y.Y.Y/Y to publish a number of web servers. 

i created a virtual IP with configuration as bellow 

config firewall vip

    edit "Mahmoud-TEST"

        set id 0

        set uuid 34e526f8-77ff-51e9-4207-cda8d9aafcdf

        set comment ''

        set type static-nat

        set extip Y.Y.Y.187

        set extintf "Main Link"

        set arp-reply enable

        set nat-source-vip disable

        set portforward disable

        set gratuitous-arp-interval 0

        set color 0

        set mappedip "172.16.16.69"

    next

the server is reachable from the FG , and can reach the FG, the WAN link works fine as i tried to browse the internet from the server using the link ( created a policy and NATed with dynamic pool from the same public subnet Y.Y.Y.Y/Y ) and everything works fine.

i also created a policy to access the server from the wan ( later i allowed All sources with All services ) but still can't reach it, i did a sniffing on the Main Link interface and find out that the packets arrives The fortigate interface but it didn't route any of them, also the PING doesn't show on the traffic log.

i tried to publish the server using the First link X.X.X.X/X and everything works fine.

Please Advise.

[Toshi_Esumi]

Technically you have three wan links to choose from to go outside (internet, I'm assuming). What kind of default route(s) do you have in the FGT? If the outside sources are on the internet you need to have a default route toward the interface to send returning packets back to the source. If don't, and if you run "debug flow", you would the common error  [size="2"]"iprope_in_check() check failed, drop"[/size] as described in below because the returning route is pointing to another interface, which would cause asymmetric routing.

https://kb.fortinet.com/kb/viewContent.do?externalId=FD31702

You need to choose from either complete equal distance (load share) or equal distance but different priorities to set all default routes.

[mahmoudmaaiteh]

well, my default route will point to the first link (x.x.x.x) which is 0.0.0.0/0 => X.X.X.1/X. 

After running debug flow, i found that the error was  "reverse path check failed, drop". after alot of digging and reading KB articles i found out that Fortigate only checks for the reverse path in the directly connected networks, static routes and dynamic routes (BGP,OSPF, etc.). and in my case the reverse path for the packets are known via policy route. so from the Reverse path check point of view, this will cause asymmetric routing which is disabled by default.

My question now is, is it fine from a security point of view to enable asymmetric routing ? because i just read that it has smth to do with IP spoofing and other security consedrations ? 

A note here, im not using asymmetric routing, because the policy route i added will make sure that the packets will leave from the same interface it arrived from.

and how can the packets bypass the path check if the reverse path is known via a policy route, not a static route ?

Thanks again,

[Toshi_Esumi]

You could eanble asym routing by "set asymroute enable" if you want to use your FGT as just a router. You would lose most of firewall features. I don't recommend it. You need to question yourself "Why did I get a firewall instead of a router???"

As in the last part of my comment above, instead you just need to have multiple default routes, in your case, with different priorities. That's all you need. Below is my home FG50E static default routes, the main one to a vlan from my ISP (C-Link) and another for a 4G Modem/router, which I use as a backup.

config router static

    edit 8         set device "ClinkVl201"         set dynamic-gateway enable      <-- because pppoe     next     edit 9         set priority 10              <--- by default the priority is 0 (highest), so set this higher than C-link         set device "wan2"         set dynamic-gateway enable    <-- because DHCP     next end

As the result, both show up in routing table "get router info routing-t all":

S*      0.0.0.0/0 [10/0] via 63.231.10.69, ppp1                       [10/0] via 100.107.137.57, wan2, [10/0] So the key is the last "10/0". "10" is the priority I set above. So all traffic/session initiated inside of this 50E always use "ppp1" interface toward C-link, of which default route has priority 0. But if any traffic from ouside comes to 100.107.137.57, like pinging from outide, the returning traffic, like ping reply, goes out of wan2 because the route is in the table. If I set VPN or VIP on wan2, it should work in the same way without making asym routing.

[mahmoudmaaiteh]

Well, 

I created a second static route as follows 

config router static edit 15 set status enable set dst 0.0.0.0 0.0.0.0 set gateway 10.100.100.1 set distance 20 set weight 0 set priority 0 set device "Main Link"   set comment '' set blackhole disable set dynamic-gateway disable set virtual-wan-link disable set dstaddr '' unset internet-service set internet-service-custom '' set link-monitor-exempt disable next end

but, it doesn't show in the routing table as yours does,

get router info routing-table all

S* 0.0.0.0/0 [10/0] via X.X.X.1, port1

there is no second entry as your case 

[Toshi_Esumi]

Look at my config and compare it with yours. I never set "distance" but "priority". The priority is the one to make the secondary, tertiary route show up as well as the primary because the distance for those routes are the same.

[mahmoudmaaiteh]

Well, you really saved my day now with the three default routes with different priorities and same distance , everything works fine and i can sleep well Thank you sir. But im little bit disappointed because reverse path check never looks in the policy routes to determine the reverse path,,, But my issue now has been resolved

[Toshi_Esumi]

To make a policy route work, there needs to be a proper route for the traffic through the interface. You never had route before through that interface that's why it failed. It's by design, not a problem.

[Toshi_Esumi]

I probably misstated. Once a session has been established, I believe it would try sending returning packets back to the interface the original packet came in regardless what kind of policy route has been set. But there was no route then it dropped before a session is established. Simply needed a route to send returning packets to.