I just overcame an interesting issue last night with a remote office. Unit is a FGT-60C running 4.3.18 with no significant changes made for a long time.
The relevant setup details:
[ul]
Domain VLAN and Guest VLAN trunked to one switch port on FGT.3 WAN lines (1 x Ethernet with own subnet/static IP, 2 x PPPoE DSL with dynamically assigned static IPs)WAN lines are in an Interface groupOn the Guest VLAN, there are several Policy Routes that direct traffic from specific IP addresses out to a particular interface and gateway. (This is so video conferencing uses a particular line, and so that mobile phone femtocells each get a dedicated DSL line to operate on.)[/ul]
What was the problem?
[ul]All traffic on the Guest VLAN ceased to work. Clients could connect to the Wi-Fi (besides the video conferencing and mobile phone femtocells, all clients on the Guest VLAN are wireless) and would get assigned an IP address via DHCP. Clients could even resolve DNS (DNS points to FGT interface address and FGT then resolves using system DNS servers).[/ul]
What did I initially think was the issue?
[ul]Problem began around the time of a major firmware update to the wireless AP in the office, so my initial assumption was that something had gone wrong with the AP, however, the same firmware on another AP in other office was working perfectly.I exhaustively troubleshot the AP (keep in mind, this was remotely so reliant on non-tech users for feedback), but could find no issue. I started to think it was hardware until I realized the video conferencing and femtocells were also down. (The femtocells are great for troubleshooting because I can view them making a connection easily on the FGT, and they have a single light that is green, yellow or red depending on the status so also easy for users to report back to me.)[/ul]
What was the actual issue and fix?
[ul]The ISP that provides the 2 x PPPoE DSL lines changed the default gateway without any announcement. (I have monitors in place to make sure the interfaces are up from the outside, but the static IP addresses themselves did not change.)After combing through the settings on the FGT, I found the default gateways didn't match in the two policy routes that used the PPPoE DSL lines.I updated the default gateway on the policy routes to match what was showing in interface settings, and traffic immediately started to flow normally again.It may be the specific gateway change contributed to this, so I will post them (xxx is the same on all): Old settings: DSL1 - IP: xxx.28.118.230 / Gateway: xxx.28.118.254 DSL2 - IP: xxx.28.82.254 / Gateway: xxx.28.82.254 New gateway settings: DSL1 - IP: xxx.28.118.230 / Gateway: xxx.28.124.253 DSL2 - IP: xxx.28.82.254 / Gateway: xxx.28.124.253 So as you can see above, the gateway became the same for both DSL lines, and the ISP enlarged the size of the subnet from a /24 to perhaps a /18.[/ul]Why was this worth posting?
[ul]It does not seem to me that 'normal' behaviour if a policy route for a single IP address has a bad gateway address that all traffic traversing the interface dies. Is this a bug?Where I got really lucky here was that I do not use policy routes to the WAN on the Domain VLAN. If that had died on me, I would have had no way to even look at this problem remotely, and I don't know how I would have ever concluded that was the issue if I didn't see it for myself! One of the reasons I maintain multiple Internet connections is so I always have a way in! There were already some other reasons I wanted to undo the WAN zone and manage each connection with separate policies/rules, but this now seals the deal for me (as I am assuming if they were separate I wouldn't have lost traffic across WAN1 which was not PPPoE DSL and had no change that should have caused it to not pass traffic). Next time I am at this remote office, I am going to work a weekend and separate the connections out![/ul]
