Policy Route not automatically switching

we have a couple of isp. Incase ISP1 fails the traffic doesnt switch automatically to ISP2. We will have to manually move the policy route above the ISP1 policy. Why is this happening. We have 4 ISP's on the box and some 80 vlans. Everytime we are moving the policy route incase any isp fails.