Multiple VLAN Trunking causing 100% CPU spike on 60E

[iamabes]

I'm hoping someone will be able to offer me some advice please, I have an issue configuring VLAN trunking, which when I enable the config I think should work, the CPU runs up to 100% on my Fortigate 60E and the device becomes unresponsive.

Very Basic topology looks like this:

Wired Users --> Zyxel GS1900-48HP --> Fortigate 60E --> Internet

Wireless Users connect first to a Cisco WAP121, which is connected to the Zyxel switch above.

Current setup (Fortigate 60E):

I have created a Hardware Switch called 'Inter-VLAN' in the Network | Interfaces section, and added interfaces 2 to 7 as members. 

I then created six VLAN sub-interfaces under the 'Inter-VLAN' switch, with VLAN IDs 20, 30, 40, 50, 60, and 70.  Each with it's own subnet, DHCP server, and DNS server.

Current setup (Zyxel GS1900-48HP):

Access ports are configured with the appropriate PVIDs, and are marked as 'un-tagged' on the appropriate VLAN ID.

Port 11 is configured as a Trunk, has a PVID of 1, is un-tagged for VLAN1, and tagged for VLANs 20, 30, 40, 50, 60 and 70.

Port 11 connects to interface 2 on the Fortigate 60E.

The current setup described above works, all end points are receiving their appropriate addresses from their respective DHCP servers, and are connecting through the SD-WAN to the internet correctly, all IPv4 policies are working as they should.

The problem occurs when I try to balance out the load of traffic from the Zyxel to the Fortigate by configuring additional trunk ports.  For example, if I configure port 13 on the Zyxel in the exact same way as port 11 (the current trunk), and connect to port 3 on the Fortigate, everything stops working.  The fortigate becomes unresponsive almost immediately, no internet access, and I can't access the Zyxel switch either.

I have tried many combinations of PVID assignment, VLAN tagging, and it seems no matter what I try, I keep getting the same result.  

Does anyone have any thoughts , questions or suggestions?

Thank you for your patience on this, I am migrating to this solution having previously used a Cisco 897VA (which incidentally had one VLAN assigned per physical port, and the Zyxel connected to it with dedicated trunk ports each tagged for a single VLAN).

[bmduncan34 · Accepted solution]

I'll bet you've introduced a loop in your network.  I've seen that in my own environment and cpu on the gate going to 100% is one indicator of that.  Are you certain you aren't introducing a spanning tree problem with port 13?

[ede_pfau · Accepted solution]

where to start...welcome to the forums!

Your original setup shortened VLAN 1 (at least) which was connected to both switch ports and at the same time to 2 FGT switch ports. IMHO, configuring a hardware switch was not favorable in the first place. But you've found that out by yourself, great.

So, the way to go is:

- create an LACP port group ("LAG") on the switch. Allow your VLANs on this (virtual) port.

- create an LACP port on the FGT, using at least 1 port (you may add more ports later, even while running). Say, it's called "trunk".

- pull your config, no encryption.

- you'll see in section "config system interface" the the VLAN ports carry a line which specifies the physical port this VLAN is attached to (example given):

    edit "tel"
        set vdom "root"
        set ip 172.16.236.1 255.255.255.0
        set allowaccess ping
        set description "20180808 for VoIP devices"
        set role lan
        set interface "internal7"   <== this is the physical port
        set vlanid 6

    next

and

config system dhcp server
    edit 1
        set lease-time 10800
        set default-gateway 192.168.234.1
        set netmask 255.255.255.0
        set interface "internal7"   <== this is the physical port
...

All you need to do now is to change that port (here: "internal7") to the LACP port, i.e. "trunk".

It may help if you search for the hw switch port name and change all occurrences EXCEPT FOR the port definition itself.

- restore this config, the FGT will reboot

- plug the cables from the FGT hw switch ports to the Zyxel switch LAG ports

- test

You may now pull all member ports from the hw switch port, and delete it after removing all references (DHCP servers, DNS, NTP, policies, routes, whatever).

[bmduncan34 · Accepted solution]

I'll bet you've introduced a loop in your network.  I've seen that in my own environment and cpu on the gate going to 100% is one indicator of that.  Are you certain you aren't introducing a spanning tree problem with port 13?

[emnoc]

yes I have to agree. Not quite understanding what he means by balancing out traffic either.

Ken Felix

[iamabes]

Thanks guys, yes I see that it is very likely that I have created a loop.  @emnoc - apologies for the ambiguous language around balancing out traffic, I was referring to achieving a load balancing effect by using multiple trunk ports on the Zyxel to route vlan traffic to the fortigate, as opposed to routing all traffic through a single port.  In my previous configuration where I had a Cisco 897 in place, each VLAN was tagged to an individual port on the switch (6 ports in total), which then connected to a corresponding port on the Cisco.  

I am able to enable either Loop Guard, or STP on the Zyxel switch.  Would you suggest enabling STP?

Do you have any suggestions on how I should tag the traffic on the trunk ports on the Zyxel?  I was thinking I would remove the un-tagged vlan1 and tagged vlan 60 from port 11, then add tagged vlan 60 to port 13 with nothing else tagged or un-tagged on that port.

[iamabes]

Quick update for anyone interested in this topic - I enabled STP on the Zyxel and the Fortigate, and found I was able to bring additional trunk ports online without both devices crashing.  However, no traffic was actually traversing the additional ports, presumably because STP was identifying the loop and shutting down the ports.

So, my new approach is to aggregate the trunk ports into one 'LAG' port on the Zyxel and create an aggregate interface on the Fortigate.  I have had some initial success, the ports come up and appear to function, but all the VLANs are still connected to the original 'Inter-VLAN' switch, and I'm looking for an easy way to move them across to the new aggregated interface.

I read in this article that I could edit the config offline, then restore.  However, when I did that, the Fortigate came up after the reboot showing no VLANs whatsoever. 

So I'm back to square one.  I'll keep experimenting, but please if anyone has any suggestions then feel free to comment.

[rwpatterson]

An aside, backup often. Before a reboot especially. This way you would have the latest config to restore to in the event this situation presents itself. Sometimes you forget what was done between changes. This would revert back to hopefully a known working version without pulling out your hair. (If you're not already bald!)

[iamabes]

Hey Bob, yes absolutely.  I am careful with backing up configs across all my devices, and keeping a thorough log of changes and current config summary.  I was easily able to test the config I mentioned above, and then revert to the prior (working config). 

I'm just hoping to find a way to easily move the VLAN sub-interfaces from the Inter-VLAN switch over to the new Aggregated Interface I've created, without having to delete the config and recreate everything.

[ede_pfau · Accepted solution]

where to start...welcome to the forums!

Your original setup shortened VLAN 1 (at least) which was connected to both switch ports and at the same time to 2 FGT switch ports. IMHO, configuring a hardware switch was not favorable in the first place. But you've found that out by yourself, great.

So, the way to go is:

- create an LACP port group ("LAG") on the switch. Allow your VLANs on this (virtual) port.

- create an LACP port on the FGT, using at least 1 port (you may add more ports later, even while running). Say, it's called "trunk".

- pull your config, no encryption.

- you'll see in section "config system interface" the the VLAN ports carry a line which specifies the physical port this VLAN is attached to (example given):

    edit "tel"
        set vdom "root"
        set ip 172.16.236.1 255.255.255.0
        set allowaccess ping
        set description "20180808 for VoIP devices"
        set role lan
        set interface "internal7"   <== this is the physical port
        set vlanid 6

    next

and

config system dhcp server
    edit 1
        set lease-time 10800
        set default-gateway 192.168.234.1
        set netmask 255.255.255.0
        set interface "internal7"   <== this is the physical port
...

All you need to do now is to change that port (here: "internal7") to the LACP port, i.e. "trunk".

It may help if you search for the hw switch port name and change all occurrences EXCEPT FOR the port definition itself.

- restore this config, the FGT will reboot

- plug the cables from the FGT hw switch ports to the Zyxel switch LAG ports

- test

You may now pull all member ports from the hw switch port, and delete it after removing all references (DHCP servers, DNS, NTP, policies, routes, whatever).

[iamabes]

Hey Ede,

Thank you so much for the welcome, and the detailed recommendation.  I had actually tried this approach, but when I rebooted the FGT it came back up again without any VLANs at all.  

Here's what I did:

1. Created an LACP port on the Switch (LAG3)

2. Created an LACP port on the FGT, called AggInt01 (currently just has ports 2, 3, 5, 6, and 7 as members)

3. Downloaded config from the FGT, unencrypted

4. Looked for any reference to 'Inter-VLAN', and changed it to 'AggInt01' (apart from the section defining the switch itself, or Port 4 currently associated with it).  The changes were made to the VLAN sub-interfaces of the Inter-VLAN switch, that I want to move across to the AggInt01 LACP port.

5. Restored the updated config to the FGT, reboot automatically

The result is that the FGT reboots, and shows the Inter-VLAN switch, port 4 associated with it, but no VLAN sub-interfaces - as expected, good news.  It also shows AggInt01 alive and well, but also with no VLAN sub-interfaces - not expected, bad news.

I was wondering whether this might be because in the config file, based on the order in which I configured the FGT, defining the interface AggInt01 appears after the VLAN sub-interfaces?

Or maybe it's something else more obvious / fundamental that I've done wrong?

[ede_pfau · Accepted solution]

yes of course, the sequence of port definitions is crucial. I know that in a config file the VLANs sometimes appear before their physical port is defined, which is no problem if these are physical.

Move the section with the LACP port on top of all VLAN definitions, and you will get your assignments.

One more thing which might provide some insight:

- connect a serial cable (RJ45 to DB9 is provided) to your PC, using PuTTY for a serial terminal window

- while restoring the config now, watch the messages

No doubt you will actually see all the errors occurring while VLANs are defined on not-yet-existant ports.