Fortigate 70D Internal network connectivity issues.

[kst]

Hello community,

We have deployed more than 30 fortigates to many different clients (most of them are hotels/hospitality services) In all of our deployments we are using internal switch interface mode, we are creating hardware switch in which are included of 1 to 4 ports, with 4-10 vlans (depending of the customer's requirements). We always use one port of the hardware switch.

All of our deployments work fine except those that we have used FortiGate 70D (7-8 deployments) and here is the issue:

All of the internal networks, randomly, are losing internet connectivity and connectivity to other ports of fortigate.

We can Access fortigate from the internet through wan ports which  still are active, and https/ssh management is active too. But we do not have access to none of the internal networks through the wan. Even if I ping any internal ip under GUI Cli console I do not get reply. 

Connectivity is never being restored unless i disable and re-enable the internal physical port and then voilà! Internet connectivity and generally all the lost connectivity to the internal interface is restored! 

(if I disable/enable just a vlan interface nothing happens).

All of the access/core switches that we are using are HP/HP procurve/Aruba switches and many models too.

Things we have tried:

[ul]

used different interfaces of the fortigate

used differernt models of access/core switches

tried tagged/untagged on the physical interface of access/core switch

STP different configs under FGT and access/core switches

Changed the link speed from 1gbps to 100mbps of the internal network (disaster)

Created 2 tickets in support.fortinet.com portal[/ul]

This problem happens ONLY to 70D's... :(

No luck so far... I'm considering of replacing all of them with FortiGate 80E or 90E or 60E

You are my last hope...!

Thank you

[Dave_Hall]

If this was a single incident, I'd would have guessed there was a network loop.  What happens if you disconnect all the internal lan cables from the fgt and connect a laptop to one of the lan ports - does the laptop get a connection and/or access the Internet?  If yes, reconnect the lan cables (one at a time) and see if the laptop still can access anything. 

Were all the 70D configuration files based off a single base config standard? 

It would help if you listed what firmware is running on the 70Ds.

[kst]

No it is not a single incident, it happends almost to all fortigates 70D and definetly it is  not network loop...(we have checked it...and also we have checked the logs in the switches)

If I disconnect and reconnect the internal cables to the switch, yes connectivity is restored. ( I guess that the same will happend if I disconnect it from internal switch and connect them to a laptop)

All of the fortigates (200B, 40C, 110C, 30D, 60D, 90D,100D, 30E, 50E, 60E,90E, 200E) if I sum all of them are about 30-40 devices, have the same philosophy on the setup, but none of them is based on a single configuration file. All of them are set up from the scratch, with no such a problem like 70Ds.

The issue is independent of the firmware version. It started from versions 5.2.x and keeps after upgrade  to versions 5.6.x and 6.0

I haven't checked yet ( I don't know if it will make sense) if I setup from the scratch a 70D to version 6.0. I would not have problem to do that if I knew that the problem would be resolved, but I doubt it.. and it is a risk because it will be another one dissapotment for the clients....

I would prefer if fortinet could replace all of these devices with 60E's or 80E'S or 90E's

[jminard]

Did you ever get a resolution to this? We have an 81E at a client and we've been experiencing this off and on since we deployed in June 2018, usually occurring on about a weekly basis, but sometimes longer in between episodes. I can still get to the Fortigate from the WAN side, but can't ping anything on the LAN side. I see all the FSSO-auths drop because the device cannot reach the internal DCs, and then after about 10 minutes they all start coming back if we don't do anything. I haven't tried disabling the port on the Fortigate that is uplinked to the network when it is occurring to see if the brings things back online faster. I reached out to Fortinet support once, but of course they wanted me to run a bunch of diags while it's occurring. Usually by the time the client contacts us, the issue is nearly over and I cannot setup the debugs in time before it corrects itself. I've been thinking it was a network loop all this time as well, but knowing there are other people with this issue that didn't have a network loop, I'm thinking it's more a hardware issue or a bug. I've been through a few different firmware versions. We don't have any other 81E's out there, so I don't have any apples to apples comparisons, all of our other devices are 60D, 60E, 101E or 200D's and have never seen this behavior on those devices. 

[Dave_Hall]

I wouldn't count out the possibility of a network loop just yet.   You may be able to rule out certain network loops just by having a laptop or computer connected directly to a free port on the same soft/hard switch (interface) as the internal network (on the fgt).  When the network "goes down" and you cannot ping the laptop/computer IP (from your WAN connection to the fgt) have someone onsite disconnect the other cables from the soft/hard switch (interface) then see if you can ping the laptop IP.  (The DC server(s) can serve this function for the ping test if they happen to be directly connected to the fgt.)

[jminard]

Ok. Currently the only thing plugged into the internal interface of the Fortigate is an uplink to a switch. So you're saying if I do have a device plugged into the internal interface and I can't ping it when the network "goes down," but then I can ping it immediately if I disconnect the uplink cable to the switch, then that would indicate a loop? What if I can't ping the device in the other port even immediately after pulling the uplink cable, what would that mean? 

Also, we don't have STP enabled on the interface (or on our switch). If there is in fact a loop right now, would turning STP on just on the Fortigate stop it from dropping the LAN connection?

[Dave_Hall]

Disconnecting the cable(s) from the internal interface on the fgt, leaving only a laptop connected (to perform connection tests), is just a simple way for almost anyone on site can try.  It also can confirm a possible issue with the 81E if internal devices suddenly started communicating with (say) the internal DCs.  

IMO enabling STP on the switch would be a good idea, and maybe even DHCP snooping.  If this switch doesn't have a manage IP, I suggest add one to it and also bind a secondary IP on the fgt internal interface so you can connect to the switch directly from the fgt CLI.  That way if this issues happens again, you may be able to remote into the siwtch to check the status and/or error logs.

[eslamelmasri]

Had the same problem, Found out that one of my team members had put my gateway as NVR's NIC ip, After so long of searching , removed the ip conflict anf bingo problem solved.. Disconnect your Fortigate and use advanced ip scanner to find which device is taking ur Fortigate ip

[emnoc]

That sounds plausible but why only 70Ds. OP is not stating one 70D, but a host of them.

Ken Felix