ISP Uplink through FortiSwitch

[jokes54321]

We are working on replacing Aruba switches with FortiSwitches. We have HA firewalls and currently use a VLAN on the Aruba to pass the ISP link to the WAN ports on the firewalls. We've run into an issue at a couple of sites where the ISP device refuses to communicate with the FortiGate when passing through an unnumbered VLAN configured on the FortiLink connection.  If we put the Aruba back in, the WAN links can then talk to the ISP gateway again.

It's only happened at a couple of our sites, so I suspect it's specific to certain brand ISP devices. At the first site it happened at, we resolved it by moving the WAN IP to the VLAN Interface under Fortilink and eliminated the uplinks to the WAN ports. At the current site we're working on, there are hundreds of IPSec tunnels and policies tied to the WAN interfaces, so moving to a VLAN interface under  FortiLink would be a time-consuming endeavor. 

Any idea on what may be causing this? 

[Toshi_Esumi]

I know why but you need to wait somebody else to provide you the best option for you. Only I can think of is don't use FortiLink but make those FSWs standalone.

The reason is all VLANs you create on FGT-managed FSWs have to come over FortiLink interface (automatic. You can see them in GUI interface page). And, those VLANs on the FortiLink don't have L2 connections to the same VLAN ID you have on other FGT ports like WAN1, in your case.

I want to know the answer from FTNT staff as well. I was thinking to do the same for our customers but I need to wait for an answer.

Toshi

[jokes54321]

Hi Toshi,

I believe I understand what you are saying and wanted to clarify something. We're not trying to pass the VLAN on Fortilink to another port on the FortiGate, through FortiLink. We want the ISP to come in on one port of the FortiSwitch, then out another port on the FortiSwitch wired up to the WAN port on the firewall.

I received a private message from a Fortinet employee indicating WAN connections on FortiLink are not recommended and to use an unmanaged switch between ISP and FortiGate. I initially shared this belief that an unmanged switch should be used between the ISP and firewalls, until a colleague convinced me I was being irrational. His take was, we're trusting VLANs on FortiLink and firewall rules to segment sensitive portions of our network to internal users, why should we not trust it for ISP links. If there is a concern for an ISP link to be attached to a VLAN on FortiLink, shouldn't the same concern exist for end user ports. 

I'm certainly not arguing one way or the other, but I sure would be interested in hearing if this is a legitimate issue or a legacy way of thinking. 

[Toshi_Esumi]

I know what you're trying to do: WAN1 to a port on the FSW, and another X1/fortilink to the same FSW for LAN traffic, which I would have tried to do the same, just like I would do with a Cisco SW.

To pass the wan VLAN, let's say VLAN 100, from ISP router to the FSW then to WAN1 on the 100F, you must have configured the VLAN 100 on the FSW and mapped the VLAN to the two ports; port1 and port2. But when you configured it the same VLAN 100 was automatically configured/mapped on the both sides of the fortilink; X1 and one of QSFP interfaces. You can see it under "fortilink" interface on the 100F in GUI.

On the 100F, both WAN1 and fortilink has VLAN ID 100 interfaces are on but they don't talk each other. While the FSW sees the same device (100F) on both ports on the VLAN. Since the 100F is not bridging them and not creating an L2 loop, it should be fine but I'm not sure about the behavior of the FSW managed by the FGT which port it would deliver packets coming from the ISP router.

As the private message is implying, the FGT-managed FSW might not be designed to handle both WAN side and LAN side of separate VLAN on the same switch. That's why I'm suggesting if you disable the fortilink and use the same X1 port as just a 10Gig LAN port, and configure the FSW as stand-alone, then your setup should work just as any other switches.

Toshi

[bweekes]

I use a managed FortiSwitch for my WAN connection w/ HA FortiGates but it does not work if I try to connect it to the FortiGate WAN port(s).

Here is what I did:

1. Create a new VLAN for the WAN connection under the WiFi Switch Controller -> FortiSwitch VLANs
   1. Assign role as WAN
   2. Set the address
2. Assign the VLAN to a port on the FortiSwitch
3. Ensure default routing is set properly under Network -> Static Routes
4. Configure the firewall policies to allow your networks to access the new WAN VLAN interface for outgoing traffic.

Now I have one WAN connection that is available to both HA FortiGate pairs.

[Toshi_Esumi]

yes, that's what the OP did and described as "moving the WAN IP to the VLAN Interface under Fortilink".

Toshi

[jokes54321]

Yep, I have approximately 30 sites setup where I was able to successfully pass both Internet circuits through a VLAN on FortiSwitch, up to the WAN ports on the firewalls. Most of these sites have a DIA primary (plugged into a Ciena) and a broadband backup. 

For one site hosted in a datacenter (ISP is using Juniper), I had to use the method you described and put the IP directly on the VLAN interface on the Fortigate, since the WAN links wouldn't communicate with the ISP. 

For this other site, there are way too many IPSec tunnels bound to the WAN link, so we'd rather not move the IP to the VLAN interface. I suspect the ISP equipment is seeing something that is triggering a block, but I don't know how to resolve it. 

I spent several hours reading through debates on Reddit talking about using your CORE switches to pass through an Internet circuit. Many said it's common, many others said it's a security risk. The PM I received from a Fortinet employee said it's not recommended, Fortilink is meant to be a collapsed core.

I guess regardless of whether or not it should be done, I'd like to know what the ISP router is seeing that is preventing it from communicating with my WAN links when passing through a FortiSwitch.

Denny

[bweekes]

I often times find myself digging into the weeds to find answers and prove something can be accomplished or not. That's just how I'm wired. However, I have to remember to look at the big picture every now and then to ensure I'm staying on track with the overall business goals.  

Given the site requirements, the recommendations from the Fortinet employee, the research you've meticulously conducted and your pay rate, would it be more cost effective to put standalone 1xxF series FortiSwitch(es) to act as WAN switches?

-Brian

[jokes54321]

Hi Brian, if we were simply talking about one or two sites, yeah, but I have over 300 sites.  An FS108 with a 3 year support agreement is roughly $400.  For redundancy, I'd want two at each location, one for each Internet circuit, so when I multiply $400 times 600 sites, I'm looking at 240K, before taxes and shipping. 

Denny