IPSec tunnel, not routing

[zmag]

I have a new IPSec tunnel and I have control over both ends of it. Local - FG60B 4.0 MR3 Remote - FG60C 4.0 MR1 The tunnel shows successful P1 and P2, but no successful pings. My first step was to tracert to a remote host. The tracert went to the firewall as expected but then it went out the default gateway not the virtual interface bound to the tunnel. the route states : destination = 10.154.154.0/24 device = rmg_dev (virtual interface) I think that just having this route should force traffic to the virtual interface, even if the tunnel was down, so why would traffic continue to gateway of last resort?

[zmag]

The 60C is a DEV box which is why we went to the newest firmware. It looks like that may have been released today according to the " Alert Message Console" Widget. chifgt02 (root) # get router info routing-table all Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area * - candidate default S* 0.0.0.0/0 [10/0] via 216.19.237.1, port16 [10/0] via 12.178.80.241, port15, [100/0] C 10.10.10.0/24 is directly connected, port17 S 10.66.4.0/24 [10/0] is directly connected, meditech S 10.66.6.0/24 [10/0] is directly connected, meditech S 10.68.48.0/24 [10/0] via 192.168.40.1, port1 S 10.240.0.0/24 [10/0] is directly connected, meditech S 10.240.64.0/24 [10/0] is directly connected, meditech S 12.123.1.233/32 [10/0] via 12.178.80.241, port15 S 12.127.16.67/32 [10/0] via 12.178.80.241, port15 S 12.127.16.68/32 [10/0] via 12.178.80.241, port15 S 12.127.17.72/32 [10/0] via 12.178.80.241, port15 S 12.129.20.0/24 [10/0] via 216.19.237.1, port16 S 12.129.199.61/32 [10/0] via 216.19.237.1, port16 S 12.129.219.155/32 [10/0] via 216.19.237.1, port16 S 12.130.132.46/32 [10/0] via 216.19.237.1, port16 S 12.147.116.3/32 [10/0] is directly connected, hologic C 12.178.80.240/28 is directly connected, port15 S 12.178.87.2/32 [10/0] via 192.168.40.9, port1 S 38.126.166.7/32 [10/0] via 12.178.80.241, port15 S 38.126.166.10/32 [10/0] via 12.178.80.241, port15 S 38.126.166.11/32 [10/0] via 12.178.80.241, port15 S 38.128.166.8/32 [10/0] via 12.178.80.241, port15 S 63.239.162.4/32 [10/0] via 12.178.80.241, port15 S 63.241.222.0/24 [10/0] via 216.19.237.1, port16 S 64.14.91.64/26 [10/0] is directly connected, dictaphone S 65.55.88.0/24 [10/0] via 216.19.237.1, port16 S 66.92.246.210/32 [10/0] via 216.19.237.1, port16 S 68.17.74.31/32 [10/0] via 216.19.237.1, port16 S 68.17.74.198/32 [10/0] via 216.19.237.1, port16 S 69.25.46.5/32 [10/0] via 12.178.80.241, port15 S 69.25.46.7/32 [10/0] via 12.178.80.241, port15 S 69.25.46.8/32 [10/0] via 12.178.80.241, port15 S 69.25.46.10/32 [10/0] via 12.178.80.241, port15 S 69.25.46.11/32 [10/0] via 12.178.80.241, port15 S 69.25.46.17/32 [10/0] via 12.178.80.241, port15 S 69.25.46.20/32 [10/0] via 12.178.80.241, port15 S 69.60.160.0/24 [10/0] via 216.19.237.1, port16 S 72.248.115.7/32 [10/0] via 216.19.237.1, port16 S 94.245.120.64/26 [10/0] via 216.19.237.1, port16 S 128.11.42.81/32 [10/0] via 12.178.80.241, port15 S 128.11.160.0/24 [10/0] is directly connected, nehen S 128.167.135.0/24 [10/0] is directly connected, nehen S 131.239.33.20/32 [10/0] via 216.19.237.1, port16 S 161.77.40.136/32 [10/0] via 216.19.237.1, port16 S 161.77.40.142/32 [10/0] via 216.19.237.1, port16 S 162.95.80.230/32 [10/0] via 216.19.237.1, port16 S 164.120.156.14/32 [10/0] is directly connected, nehen S 192.68.48.0/22 [10/0] is directly connected, philips S 192.168.1.0/24 [10/0] is directly connected, amherst S 192.168.1.11/32 [10/0] is directly connected, amherst S 192.168.1.12/32 [10/0] is directly connected, amherst S 192.168.1.105/32 [10/0] is directly connected, amherst S 192.168.1.107/32 [10/0] is directly connected, amherst S 192.168.1.109/32 [10/0] is directly connected, amherst S 192.168.1.113/32 [10/0] is directly connected, amherst S 192.168.30.0/24 [10/0] via 192.168.40.1, port1 S 192.168.32.0/24 [10/0] via 192.168.40.1, port1 S 192.168.36.0/24 [10/0] via 192.168.40.1, port1 S 192.168.37.0/24 [10/0] via 192.168.40.1, port1 S 192.168.38.0/24 [10/0] via 192.168.40.1, port1 S 192.168.39.0/24 [10/0] via 192.168.40.1, port1 C 192.168.40.0/22 is directly connected, port1 C 192.168.99.0/24 is directly connected, port2 S 192.168.143.0/24 [10/0] via 192.168.40.1, port1 S 192.168.230.0/24 [10/0] via 192.168.40.1, port1 S 192.168.238.0/24 [10/0] via 192.168.40.1, port1 S 192.168.240.0/24 [10/0] via 192.168.40.1, port1 S 195.33.169.46/32 [10/0] via 216.19.237.1, port16 S 198.92.116.3/32 [10/0] is directly connected, dictaphone S 198.92.118.0/24 [10/0] is directly connected, dictaphone S 198.92.119.247/32 [10/0] is directly connected, dictaphone S 198.92.119.248/32 [10/0] is directly connected, dictaphone S 199.26.210.197/32 [10/0] via 12.178.80.241, port15 S 199.107.238.205/32 [10/0] via 216.19.237.1, port16 S 203.163.124.46/32 [10/0] via 216.19.237.1, port16 S 204.165.229.185/32 [10/0] via 216.19.237.1, port16 S 204.165.229.186/32 [10/0] via 216.19.237.1, port16 S 204.228.201.10/32 [10/0] via 216.19.237.1, port16 S 204.228.201.11/32 [10/0] via 216.19.237.1, port16 S 204.250.122.121/32 [10/0] via 216.19.237.1, port16 S 206.16.57.70/32 [10/0] via 216.19.237.1, port16 S 206.28.216.0/28 [10/0] is directly connected, dictaphone S 207.46.51.64/26 [10/0] via 216.19.237.1, port16 S 207.46.163.0/24 [10/0] via 216.19.237.1, port16 S 207.127.11.8/32 [10/0] via 216.19.237.1, port16 S 208.78.140.29/32 [10/0] is directly connected, company_medical S 208.78.140.30/32 [10/0] is directly connected, company_medical S 208.86.145.233/32 [10/0] via 12.178.80.241, port15 S 208.86.145.235/32 [10/0] via 12.178.80.241, port15 S 208.86.145.239/32 [10/0] via 12.178.80.241, port15 S 208.86.145.253/32 [10/0] via 12.178.80.241, port15 S 208.86.145.254/32 [10/0] via 12.178.80.241, port15 S 209.117.210.233/32 [10/0] via 12.178.80.241, port15 S 209.117.210.235/32 [10/0] via 12.178.80.241, port15 S 209.117.210.253/32 [10/0] via 12.178.80.241, port15 S 209.117.210.254/32 [10/0] via 12.178.80.241, port15 S 209.237.226.38/32 [10/0] via 216.19.237.1, port16 S 213.199.154.0/24 [10/0] via 216.19.237.1, port16 S 213.199.180.128/26 [10/0] via 216.19.237.1, port16 C 216.19.237.0/27 is directly connected, port16 S 216.32.180.0/24 [10/0] via 216.19.237.1, port16 S 216.32.181.0/24 [10/0] via 216.19.237.1, port16 S 216.57.136.43/32 [10/0] via 216.19.237.1, port16 S 216.165.132.231/32 [10/0] is directly connected, epic [10/0] via 216.19.237.1, port16 S 216.231.91.67/32 [10/0] via 216.19.237.1, port16 S 216.231.91.167/32 [10/0] via 216.19.237.1, port16

[ede_pfau]

OMG.

[ede_pfau]

I know this is not directly related to your problem but maybe the FGT has difficulties in inserting just another route with a list that long... I' d go over the routes and pack them together as much as possible. Example: amherst: it' s just 192.168.1.0/24, all host routes are included in that. 192.168.36.0/24 ... .39.0/24 is a 192.168.36.0/22 route. You mentioned that one of the FGTs is a dev box. Could you test with just the default route and the static route to the VPN in it? Might still be messy enough because of all the directly connected routes. Route insertion must work or your tunnel will not work correctly.

[zmag]

OMG ? That can' t be good.

[rwpatterson]

104 static routes? Holy cow! Have you tried any routing protocols to ease the load? RIP or OSPF? I have just about as many advertised routes, but only 12 or so are static to the FGT, the rest are learned with OSPF. Add a route somewhere else, don' t have to mess with routes in 14 boxes... [:' (] My $.02 Also, using a x.x.x.0/24 route is much cleaner. You should use policies to define what entities can traverse the VPN. This will not affect the tunnel as far as quick mode selections and such. Also grouping together multiple /24 bit subnets into supernets works as well. (such as Dictaphone, 198.92.116.0/22 will cover 198.92.116.0-198.92.119.255) One static covers four of your original routes. Simplify. Another example, 208.86.145.0/24 will cover 5 entries on port15. Simple house cleaning will make things far easier to debug.

[zmag]

Thanks for the input. I will look into that.

[zmag]

So, about my routes... I have one cluster that is the gateway out for 6 sites. One of the reasons I have so many routes is because I have 2 ISPs and without specifying routes some of my reply traffic was going out the other ISP. In other words, an outside prescription company would generate traffic inbound to me, unless I specified a route to port16 some of that traffic went to port15 and failed. My original assumption was that the traffic would reply to the source but that did not happen. I am all ears if there is a better way than adding routes, which have to be maintained as ip' s change. I went through the list and agree that I could consolidate the table with supernetting and subnetting as opposed to /32. I' ll have to draft it to see how many entries I can save. Since I only have one firewall I don' t think routing protocols will help me here. As far as this particular routing issue, I opened a ticket with FG support, so we’ll see how that goes. Thanks for the help so far, if you have more I’ll listen.

[rwpatterson]

My OSPF is between my firewall and my internal routers, not just Fortigates.

[ede_pfau]

just one more caveat: if you replace /32 host routes with /24 subnet routes and block firewall traversal with policies for the hosts you don' t want to route there is a subtle difference to your current setup. Those hosts not included in a route now will be using the default route (and make it to the internet) whereas with subnet routes plus policies they will be blocked. It depends on whether you care for those not included in your routes now or not.

[zmag]

OK, so I made my list last night and consolidated before hours today. I was only able to remove 22 routes and replace them with 4 for a net gain of 18. Before I touched any routes I decided to tear down the tunnel on both sides and rebuild it, which I did and I got the exact same results. I figure it' s probably a routing issue, so I added a new route that states 151.203.191.204 gtw= 12.178.80.241 port15 this is a public ip in my dev block with no host attached but I assume the route should still go to port 15 and then fail. It did not, it went to port16 (default route) So either I misunderstand or there is a routing issue. Then I consolidated the 18 routes and tried again. Here is my worksheet; added 192.168.36.0/255.255.252.0 (/22) port1 192.168.40.1 deleted 36,37,38 and 39/24 added 69.25.46.0/255.255.255.224 (/27) port15 12.178.80.241 SureScripts deleted 5,7,8,10,11,17 and 20 verified added 192.168.1.0/255.255.255.0 /24) amherst deleted 11,12,103,105,107,109 added 208.86.145.224/255.255.255.224 (/27) port15 12.178.80.241 SureScripts deleted 233,235,253,254,239 After each was done I tested with a tracert and found something I didnt expect. Any tracert to a valid host (accepting icmp) goes to the correct path, hosts that dont respond to icmp go to the default route. Again, I think the route should force all traffic to the defined path/gateway regardless of whats on the other side. I re-ran --- get router info routing-table all ---and the table hasnt updated any of the changes made today. I' m starting to think something may be up with the routing service. I will raise the level of my support ticket next but I thought you guys might have some more helpful info. And amazingly still no production outages Thanks,