FortiGate HA Design with Standalone Switches

[67vwbug]

Greetings all,

I have a question I hope you all can help me out with regarding FortiGate HA network design with standalone switches as I am experiencing some intermittent network issues on the internal LAN.  I am starting to wonder if it could be a design issue.  This is my first exposure to FortiGate firewalls and all other environments I have worked in have also had stacked switches instead of standalone.  This design serves a small rack of servers at a remote site and was architected to eliminate as many single points of failure as possible.  The internal switches tie into Hyper-V hosts configured for Switch Independent teaming.

In addition to firewalling, the FortiGate is also providing routing at this site.  Ports 1 and 2 of the FortiGates are configured as a hardware switch and trunked to the internal switches.  Interfaces are then configured for VLANs for the various internal networks.

Design:

Observations from the network at this site are:

[ul]

Port 2 from the passive FortiGate to the internal switch is always in STP blocking mode, suggesting the presence of a loop to me.

High occurrences of TCP retransmits have also been observed during backup jobs, which can be eliminated by "pinning" the backup server and client (both on the same L2 network) to one of the internal switches instead of networking through the Hyper-V team.

The SAN at this site also alerts to losing connection to the DNS server occasionally even though the DNS server is online at the time of occurrence.

Microsoft Windows clustering reports of network fencing on non-routable (private, not trunked up to the FortiGates) VLANs, such as used for internal cluster communication, when two nodes in the cluster are on different physical Hyper-V hosts.[/ul]

The Hyper-V network configuration has been reviewed numerous times and we believe to be configured to best practices.  Syslog and monitoring of the network environment has so far not helped to yield any root cause.

While looking through Fortinet documentation I came across an example of a full mesh HA configuration (http://help.fortinet.com/fos50hlp/54/Content/FortiOS/fortigate-high-availability-52/HA_full_meshExam...) and it made me wonder if the current environment should be rearchitected to look more like this than the current architecture.

Questions:

1.  Is there anything you would change design-wise in this case?

2.  Is there anything you could think of networking or logging-wise to further test to try to further pinpoint the issue?

3.  Could the standalone switches be part of the problem?  Should we be looking to replace them with stacked switches?

4.  The FortiGate interfaces are not configured as redundant interfaces as in the full mesh example.  Could this be part of the problem and can these interfaces be changed easily or will it require extensive reconfiguration of the firewalls?

All I can think of for now.

Thanks in advance,

JR

[67vwbug]

Any thoughts?  Or anybody willing to share the Fortinet HA architecture that you use with standalone switches?

I'm tempted to try a full mesh design with redundant interfaces during a downtime over the holidays.  Is there an easy way to change interfaces from LAN to Redundant (perhaps via the CLI), or am I going to need to recreate the network interfaces and all policy rules, etc. that are dependent on them?

[btp]

67vwbug wrote:

Is there an easy way to change interfaces from LAN to Redundant (perhaps via the CLI), or am I going to need to recreate the network interfaces and all policy rules, etc. that are dependent on them?

Fortinet has yet to appreciate the use of CHANGES in a setup. You can't even change VLAN-ID on an interface. And the reference to said interface must be deleted before you can delet the interface. It is wise to use zones, and point policies to these zones instead of the interface/vlan itself - even if there is only one interface/vlan in this zone. Then you have some more flexibility.

With regards to your design - this is how we have set up HA clusters. We use BGP between the FG and the PE. You do have a loop on the inside, since your internal network spans both switches. Also, in my opinion, you do have a full-mesh network here. We have the exact same setup at one customer, and had to use 100D to be able to use redundant interfaces. In my opinion this is much more elegant than STP. Not sure if STP is enabled by default on the FG.

[MikePruett]

With teaming and what not implemented you are most likely going to want some aggregate connections on the Gate as well as some port channels on the switches (if they allow it)....I would probably look at VSS Pairing the switches if they are cisco brand.