Problem with 2nd network reaching internet

[dcs]

I'm definitely a beginner with Fortigate, but have some experience with stateful firewalls (from iptables). I'm setting up a system (Fortigate 30E) which should look like:

SD-WAN (wan + lan4) -> two different internet connections, select ports forwarded to Internal (below)

Local (lan3): Single computer that is a local server and responsible for most of our load (including stuff it forwards), unlimited outgoing connections via SD-WAN.

Lan (lan1 & lan2): connect to some WiFi routers; unlimited outgoing connections via SD-WAN, limited to select ports to Local.

My Configuration so far is outlined below. Note that I know I do not have rules to permit LAN to access anything on Local right now.

What works: Internet connections to Local for selected ports only; Local accessing internet; Load balancing with SD-WAN.

What Doesn't work: Lan accessing internet. Using the packet sniffer, I determined that for DNS requests going out from Lan, the returning packets from the nameserver (I tested with 8.8.8.8) are rejected by the Fortigate, with an ICMP response sent to the DNS server that the destination port is not accessible.

The same traffic from Local seems to work just fine.

Any thoughts on what I've done wrong, or how to identify what I've done wrong would be greatly appreciated.

Thanks,

Doug

(Names and addresses have been sanitized, and the number of ports reduced).

Network: lan (lan1 & lan2) Role LAN Addressing Manual nnn.nnn..1.99/24 DHCP server starting nnn.nnn.1.100 ending nnn.nnn.1.200 STP on local (lan3) Role LAN Addressing Manual nnn.nnn..2.1/29 DHCP server starting & ending at nnn.nnn.2.2 SD-WAN combines two interfaces wan2 (lan4) Role WAN Addressing Manual XXX.XXX.XXX.XXX to ISP1 (gateway configure in SD-WAN page) wan Role WAN Addressing Manual YYY.YYY.YYY.YYY to ISP2 (gateway configure in SD-WAN page) Static routes: 0.0.0.0/0            SD-WAN nnn.nnn.0.0/16    0.0.0.0        local (lan3) nnn.nnn.1.0/24    0.0.0.0        lan Virtual IPs: ip1: XXX.XXX.XXX.XXX tcp port 20 to nnn.nnn.2.2, interface any ip2: YYY.YYY.YYY.YYY tcp port 20 to nnn.nnn.2.2, interface any ip3: XXX.XXX.XXX.XXX tcp port 22 to nnn.nnn.2.2, interface any ip4: YYY.YYY.YYY.YYY tcp port 22 to nnn.nnn.2.2, interface any Virtual IP Groups ipg1: Interface lan4       Members ip1 and 1p3 ipg2: Interface wan       Members ip2 and 1p4 IP Pool (not used): nnn.nnn.2.100-nnn.nnn.2.200 Policies: 0: Implicit Deny (built in) 1: To Local  sd-wan -> lan3    Source all    Dest ipg1, ipg2    Schedule Always    Service All    Accept    NAT Disabled 2: To Internet lan3 -> sd-wan    Source all    Dest all    Schedule always    Service all    NAT Enabled    Use Outgoing Interface Address 3: To Internet2 lan -> sd-wan    Source all    Dest all    Schedule always    Service all    NAT Enabled    Use Outgoing Interface Address Observed lan -> sd-wan fails because (at least) DNS responses are rejected by Fortigate (which sends ICMP: Port Not Accessible to DNS server).

[orani · Accepted solution]

Try to place the rule with id 2 before (higher) the rule with id 1

[orani · Accepted solution]

On the upper right corner check the sequence view. Then check you rules priority. The rule wich is higher is the first to execute... then you can drag and drop the rule to move upper

[orani]

The implicit deny rule must be the last rule you have.

[dcs]

Orani,

Thanks for your reply. Since the deny rule is implicit, it's position is determined by the Fortigate software. I imagine if you check your rules, you will find the implicit rule (if you enable seeing it) shows up as rule 0 also. Also, if it really were first, nothing would work at all. As it is most things work, just not packets returning from the internet to the second internal interface.

Regards,

Doug

[orani · Accepted solution]

Try to place the rule with id 2 before (higher) the rule with id 1

[dcs]

orani,

Thanks for the quick reply. I haven't been ordering the rules myself. The numbering is generated by the FortiGate GUI, and the actual order they occur in that interface is based on the interfaces involved. How do I

1) Determine the actual order of priority of the rules, and

2) Control that order.

Regards,

Doug

[orani · Accepted solution]

On the upper right corner check the sequence view. Then check you rules priority. The rule wich is higher is the first to execute... then you can drag and drop the rule to move upper

[dcs]

orani,

Thank you. I tried to drag before, but it was in interface pair view and would not let me. Now I see the priority (from highest to lowest) is 1, 2, 3, 0. My problem is with traffic from rule 3; rule 2 traffic works correctly. Should I move rule 3 above 1? Also, it sounds like the VIP in rule 1 takes effect first even if rule 1 is last -- but that VIP is restricted to ports which are not related to the problem, so I'm hoping it doesn't matter.

Doug

[Dave_Hall · Accepted solution]

Try removing the static routes for local and lan - the networks tied to these interfaces should already be in the routing table (but don't do this unless you are on site or have alternate access to get into the fgt.)

If it's just DNS-related only issue, check the WAN interfaces to see if DNS override is enabled or not. 

If would help if you provide the packet sniffer output and/or show us the routing monitor output.

Edit: Have you confirmed or isolated the issue on the client side behind the lan network and/or confirmed the information handed out by the DHCP server to devices on that lan is correct?

dcs wrote:

Static routes: 0.0.0.0/0            SD-WAN nnn.nnn.0.0/16    0.0.0.0        local (lan3) nnn.nnn.1.0/24    0.0.0.0        lan

[dcs]

Dave,

I've already tried without the static rules, and (because of our internal network topology) they are required.

I don't see DNS override on either interface.

I'll check DHCP and edit the packet logs to sanitize them so I can post them after lunch.

Thanks for the tips,

Doug

[dcs]

Here are the packet logs. I didn't filter for the Fortinet DNS servers, so you don't see the ICMP replies to them, or the DNS responses coming from them. However, in another trace I did see the same thing for those as I see for the google name server here. Note that only DNS is captured here, because everything else is waiting on name resolution.

<Serial Number> # diag sniffer packet any "host nnn.nnn.1.100 or host 8.8.8.8" 4 20 a interfaces=[any] filters=[host nnn.nnn.1.100 or host 8.8.8.8] 2019-08-16 19:00:42.452130 lan in nnn.nnn.1.100.48860 -> 208.91.112.53.53: udp 33 2019-08-16 19:00:42.452193 lan in nnn.nnn.1.100.48860 -> 208.91.112.52.53: udp 33 2019-08-16 19:00:42.452221 lan in nnn.nnn.1.100.48860 -> 8.8.8.8.53: udp 33 2019-08-16 19:00:42.452243 lan4 out YYY.YYY.YYY.YYY.48860 -> 8.8.8.8.53: udp 33 2019-08-16 19:00:42.452247 eth0 out YYY.YYY.YYY.YYY.48860 -> 8.8.8.8.53: udp 33 2019-08-16 19:00:42.452253 lan in nnn.nnn.1.100.48860 -> 208.91.112.53.53: udp 33 2019-08-16 19:00:42.452511 lan in nnn.nnn.1.100.48860 -> 208.91.112.52.53: udp 33 2019-08-16 19:00:42.452526 lan in nnn.nnn.1.100.48860 -> 8.8.8.8.53: udp 33 2019-08-16 19:00:42.452535 lan4 out YYY.YYY.YYY.YYY.48860 -> 8.8.8.8.53: udp 33 2019-08-16 19:00:42.452539 eth0 out YYY.YYY.YYY.YYY.48860 -> 8.8.8.8.53: udp 33 2019-08-16 19:00:42.459044 lan4 in 8.8.8.8.53 -> YYY.YYY.YYY.YYY.48860: udp 49 2019-08-16 19:00:42.459081 lan4 out YYY.YYY.YYY.YYY -> 8.8.8.8: icmp: YYY.YYY.YYY.YYY udp port 48860 unreachable 2019-08-16 19:00:42.459086 eth0 out YYY.YYY.YYY.YYY -> 8.8.8.8: icmp: YYY.YYY.YYY.YYY udp port 48860 unreachable 2019-08-16 19:00:42.469131 lan4 in 8.8.8.8.53 -> YYY.YYY.YYY.YYY.48860: udp 93 2019-08-16 19:00:42.469151 lan4 out YYY.YYY.YYY.YYY -> 8.8.8.8: icmp: YYY.YYY.YYY.YYY udp port 48860 unreachable 2019-08-16 19:00:42.469155 eth0 out YYY.YYY.YYY.YYY -> 8.8.8.8: icmp: YYY.YYY.YYY.YYY udp port 48860 unreachable 2019-08-16 19:00:43.013578 lan in nnn.nnn.1.100.34520 -> 8.8.8.8.53: udp 39 2019-08-16 19:00:43.013619 lan4 out YYY.YYY.YYY.YYY.34520 -> 8.8.8.8.53: udp 39 2019-08-16 19:00:43.013623 eth0 out YYY.YYY.YYY.YYY.34520 -> 8.8.8.8.53: udp 39 2019-08-16 19:00:43.014352 lan in nnn.nnn.1.100.48860 -> 208.91.112.53.53: udp 39