Wrong routing in redundant VPN

[kckong]

Hello, I have managing 10 fortigate units in 10 offices, they are 2x FGT-60 in 3.0 MR7 patch 9, 5x FGT-60B in 4.0 MR1 patch 4, and 3x FGT-100A in 4.0 MR1 patch 4. In head office, the FGT-100A have 2 WAN links, each link with fixed IP, to form a redundant VPN (running in IPSEC interface mode, have correct routing and policies) and load balance internet access for staff in head office. All branch office (have 1 WAN link with fixed IP) connect to head office e-mail server (using the e-mail server internal IP address) for sending & receiving e-mail, and file sharing. I found that in above said config, all FGT-60B are running smoothly. (i.e., if one of the VPN link go down, branch office still can assess head office via another VPN link) However, I found that in one of the office that using FGT-60, some how one of the PC in that office cannot access e-mail server in the head office. When I remote access that PC via VNC (I still can access this PC via VNC, with that PC internal IP when problem occurs) and do a trace route test. I found that the fortigate view the e-mail server internal IP as WAN IP, and route that IP to the WAN side. However, at that PC when I do another test, i.e., by trace route / ping to another IP in head office, the route result is correct. Moreover, the above said problem is not happens on one dedicated PC, ie. the problem may happens on PC A this time, an then on another PC next time. I have no idea what is the problem, as all FGT units using same VPN settings. The attachmet is the result of the trace route test. In the test, 10.10.10.108 is a PC, and 10.10.10.24 is the e-mail server, all these two IP are behind the FGT-100A(10.10.10.4) in head office. Beside ask for help on this forum, I have also submit a support ticket, and waiting for their feedback. Thanks Danny

[rwpatterson]

Do you have any strange static or policy routes on that unit?

[kckong]

Hello, No. I only use static route, no policy route, please refer to the screen dump. The settings of these 10 FGT units are similar, almost the same. The differnet between them are the LAN & WAN IP, URL block list, routing & policies are similar. Actually, I also have one FGT-60 at home, and also had the same problem. So, I worry it is a bug of 3.0 MR7 Patch 9 Thanks Danny

[red_adair]

So what you have is 2 Routes to the same Destination, same Metric, over 2 different Interfaces. How can poot FortiOS know which path to take ? (Actually with MR7 they introcuded SRC-based ECMP, so depending off which PC you test you see different results, means different routes the packets take). There is a CLI command in the static route conf route static edit <id of your sec route> set priority 1-100 This basically decides gives the backup Static Route a different Admin-Prio, hence it will not get into decision. (Though it will stay in routing monitor). v-2 You also simply could have changed the Metric. This only leaves the lower Distance in the Routing Table. -R.

[rwpatterson]

You could also use a routing protocol (like RIP or OSPF) to automatically choose which route is best.

[kckong]

Thank you your information. For the redundant VPN settings, I just follow the article form the Fortigate knowledge base. I will study OSPF & RIP, and then improve the routing. Thanks Danny

[MasterBratac]

I´m having very similar problems like that at one of our customers networks. Head Office was 100A, now 80C, soon 110C Branch Offices are around 10 60B and some 50B and 50B Wifi. Head office has 3 WAN Connections. One is used for Internet Access only, the other two are used for redundant VPN connections (One of them has lower metric in the routing table). Sometimes a single host of the branch offices cannot acces a single host in the head office ... all other traffic is OK. That means this host is able to access other hosts in the head office, and the host in the head office is also accessable by other hosts. I´ve this problems a long time now over many firmware Versions. Fortinet support was´nt able to help :-( If I´m investigating the packetflow on the head offices FGT, I could see, that the packet of the branch host are comming in, but the answers where routet to the wrong interface. If I shut down this wrong interface and reenable it, everything is OK again for a few weeks. Weird ...

[abelio]

Hello,

Head office has 3 WAN Connections. One is used for Internet Access only, the other two are used for redundant VPN connections (One of them has lower metric in the routing table). .. Weird ...

Did you tried this approach? Let suppose that your remote network accesible by VPN be 10.10.10.0/24 Assuming you' re working with route based redundant VPNs, you' ll have at least 3 static routes to remote networks:

 config router static
     edit 1
         set device " PortX" 
         set distance 10
         set dst 0.0.0.0 0.0.0.0
     next
     edit 2
         set device " mainVPN" 
         set distance 5
         set dst 10.10.10.0 255.255.255.0
     next
     edit 3
         set device " redundantVPN" 
         set distance 6
         set dst 10.10.10.0 255.255.255.0
       next
 end
 

I noted that sometimes (specially with dynamic addressed wans), problem arises when mainVPN goes down, and redundantVPN doesn' t take over quickly and routing table is not correctly updated. The symptom is that traffic to remote network 10.10.10.x became to flow to the default internet route. This setting apparently solved the issue: add a blackhole route with lower metric than default one but higher than VPN' s ones; in above example terms it would be:

 config router static
     edit 4
         set blackhole enable
         set distance 7
         set dst 10.10.10.0 255.255.255.0
     next
 end
 

This avoid that traffic goes to the internet while vpns can do its work. hope it helps.

[MasterBratac]

Hello Abelio, at the moment it´s configured exactliy like in your example. But I think this blackhole route sound like a good Idea! I wil give it a try. Thankyou!

[kckong]

Hello, I have test OSPF and found that using such method for redundant routing is much faster than static route. Back to my question, I change back to policy VPN temporary, as I found that one of the WAN line have problem, and the ISP is checking the issue.