OK, first, I' m glad you got your setup working now.
On second thought, what happened here?
To achieve your goal, you don' t have to have both default routes showing up at the same time in the Routing Table. If you configure two def. routes with exactly the same distance and priorities then the FGT enables ECMP (Equal Cost, Multiple Paths routing) which gives you 2
active default routes for load balancing and/or redundancy.
Just to achieve routing you only need one default route (to the other local gateway). If you configure 2, you get redundancy for free: if the main gateway fails (downtime etc.), traffic will be re-routed to WAN2, keeping your internet access alive. When the first default route is determined down, it will be dropped from the Routing Table, and the (hidden) second default route will be added and used.
But with ECMP active, having traffic sent out wan1 for even source IP addresses and out wan2 for the odd ones is NOT what you would like to have. But that' s what the FGT will do to load-balance the traffic onto two equal routes.
So, in order to " correct" this (you will have noticed the alternating behavior), you introduced Policy Routes. Policy Routing bypasses all of the Routing Table and thus the faulty setup is " corrected" i.e. overridden.
Just to clarify: regular routes determine the egress interface by the
destination address (and only the destination address!). Policy Routes can look at the
source address, source port or destination port as well to determine the gateway.
So, Best Practice demands to use the least complicated mechanism to achieve the goal, and in your case, you only need to separate traffic by it' s destination address. That' s why your setup should work with regular routes right from the start.
Now back from theory to real life: apparently, in your case the configuration doesn' t work. The dependence on " distance=1" is awkward and leads me to believe that this is due to a bug in FOS 5.x. (If someone has a little time on his hands it would be great to test this on a 5.x FGT).
So, actually, the way you have it set up now might be the only way it will work, and you can well leave it at that.
Ede Kernel panic: Aiee, killing interrupt handler!