Trunk between Cisco switch and Fortigate using LACP

[teksolutions]

Trying to get a trunk built between a Cisco Catalyst switch and a Forigate 100F using two 10G links in an LCAP link-aggregation configuration.  The LACP link comes up but the VLAN communication does not work.  I swear I've used this same configuration in the past and it worked, but it isn't working now.

Here is the configuration on the Fortigate:

config system interface
    edit "x1"
        set vdom "root"
        set type physical
        set trunk enable
        set snmp-index 7
    next
    edit "x2"
        set vdom "root"
        set type physical
        set trunk enable
        set snmp-index 8
     next
     edit "po1"
        set vdom "root"
        set allowaccess ping https ssh snmp http fgfm radius-acct fabric ftm
        set vlanforward enable
        set type aggregate
        set member "x1" "x2"
        set alias "Trunk to Cisco"
        set device-identification enable
        set lldp-transmission enable
        set role lan
        set snmp-index 100
        config ipv6
            set ip6-allowaccess ping
        end
    next

    edit "po1.vlan10"
        set vdom "root"
        set device-identification enable
        set role lan
        set snmp-index 49
        set interface "po1"
        set vlanid 10
    next
end

Here is the configuration on the Cisco

interface Port-channel1
    description Trunk to Fortigate
    switchport trunk native vlan 10
    switchport mode trunk
    spanning-tree portfast trunk
!
    interface TenGigabitEthernet1/1/1
    switchport trunk native vlan 10
    switchport mode trunk
    logging event bundle-status
    channel-protocol lacp
    channel-group 1 mode active
!
interface TenGigabitEthernet1/1/2
    switchport trunk native vlan 10
    switchport mode trunk
    logging event bundle-status
    channel-protocol lacp
    channel-group 1 mode active
!
interface Vlan10
    ip address 192.168.10.2 255.255.255.0
!

[teksolutions]

oh here is the LACP diags on the Cisco, not sure how to do the same for Fortigate...

SW1#sh lacp neighbor
Flags: S - Device is requesting Slow LACPDUs
F - Device is requesting Fast LACPDUs
A - Device is in Active mode P - Device is in Passive mode

Channel group 1 neighbors

LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Te1/1/1 SA 255 e023.ff67.a6dc 0s 0x0 0x21 0x2 0x3D
Te1/1/2 SA 255 e023.ff67.a6dc 29s 0x0 0x21 0x1 0x3D

SW1#sh etherchannel detail
Channel-group listing:
----------------------

Group: 1
----------
Group state = L2
Ports: 2 Maxports = 16
Port-channels: 1 Max Port-channels = 16
Protocol: LACP
Minimum Links: 0

Ports in the group:
-------------------
Port: Te1/1/1
------------

Port state = Up Mstr Assoc In-Bndl
Channel group = 1 Mode = Active Gcchange = -
Port-channel = Po1 GC = - Pseudo port-channel = Po1
Port index = 0 Load = 0x00 Protocol = LACP

Flags: S - Device is sending Slow LACPDUs F - Device is sending fast LACPDUs.
A - Device is in active mode. P - Device is in passive mode.

Local information:
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Te1/1/1 SA bndl 32768 0x1 0x1 0x133 0x3D

Partner's information:

LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Te1/1/1 SA 255 e023.ff67.a6dc 4s 0x0 0x21 0x2 0x3D

Age of the port in the current state: 0d:20h:10m:40s

Port: Te1/1/2
------------

Port state = Up Mstr Assoc In-Bndl
Channel group = 1 Mode = Active Gcchange = -
Port-channel = Po1 GC = - Pseudo port-channel = Po1
Port index = 0 Load = 0x00 Protocol = LACP

Flags: S - Device is sending Slow LACPDUs F - Device is sending fast LACPDUs.
A - Device is in active mode. P - Device is in passive mode.

Local information:
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Te1/1/2 SA bndl 32768 0x1 0x1 0x134 0x3D

Partner's information:

LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Te1/1/2 SA 255 e023.ff67.a6dc 3s 0x0 0x21 0x1 0x3D

Age of the port in the current state: 0d:20h:09m:52s

Port-channels in the group:
---------------------------

Port-channel: Po1 (Primary Aggregator)

[teksolutions]

Actually, here is some diag info I found from the Fortinet:

fw # diag netlink aggregate list
List of 802.3ad link aggregation interfaces:
1 name fortilink status up algorithm L4 lacp-mode static
2 name po1 status up algorithm L4 lacp-mode active

fw # diag netlink interface list po1

if=po1 family=00 type=1 index=62 mtu=1500 link=0 master=0
ref=31 state=start present fw_flags=b800 flags=up broadcast run master multicast
Qdisc=noqueue hw_addr=e0:23:ff:67:a6:dc broadcast_addr=ff:ff:ff:ff:ff:ff
stat: rxp=498796 txp=9781 rxb=99075260 txb=1719506 rxe=0 txe=0 rxd=0 txd=0 mc=412950 collision=0 @ time=1704487183
re: rxl=0 rxo=0 rxc=0 rxf=0 rxfi=0 rxm=0
te: txa=0 txc=0 txfi=0 txh=0 txw=0
misc rxc=0 txc=0
input_type=0 state=3 arp_entry=0 refcnt=31

fw # diag netlink interface list x1

if=x1 family=00 type=1 index=23 mtu=1500 link=0 master=62
ref=26 state=start present fw_flags=0 flags=up broadcast run allmulti slave multicast
Qdisc=mq hw_addr=e0:23:ff:67:a6:dc broadcast_addr=ff:ff:ff:ff:ff:ff
stat: rxp=455704 txp=4883 rxb=67212019 txb=859227 rxe=0 txe=0 rxd=0 txd=0 mc=405869 collision=0 @ time=1704487204
re: rxl=0 rxo=0 rxc=0 rxf=0 rxfi=0 rxm=0
te: txa=0 txc=0 txfi=0 txh=0 txw=0
misc rxc=0 txc=0
input_type=0 state=3 arp_entry=0 refcnt=26

fw # diag netlink interface list x2

if=x2 family=00 type=1 index=24 mtu=1500 link=0 master=62
ref=26 state=start present fw_flags=0 flags=up broadcast run allmulti slave multicast
Qdisc=mq hw_addr=e0:23:ff:67:a6:dc broadcast_addr=ff:ff:ff:ff:ff:ff
stat: rxp=43244 txp=4902 rxb=31894635 txb=860987 rxe=0 txe=0 rxd=0 txd=0 mc=7199 collision=0 @ time=1704487206
re: rxl=0 rxo=0 rxc=0 rxf=0 rxfi=0 rxm=0
te: txa=0 txc=0 txfi=0 txh=0 txw=0
misc rxc=0 txc=0
input_type=0 state=3 arp_entry=0 refcnt=26

[Toshi_Esumi]

You have both "fortilink" and "pol1" LAGs up. Do you happen to have any FGT-managed Fortiswitch(es) in addition to the Catalyst?

Toshi

[teksolutions]

The Fortilink LAG is for the FortiSwitch-224E which is being replaced with the Cisco (recommend that no one ever buys a FortiSwitch)

[Toshi_Esumi]

Are you aware that FGT uses/reserves VLAN ID 1 for fortilink for FSW management?
https://community.fortinet.com/t5/FortiGate/Technical-Tip-Reserved-VLAN-ID-1/ta-p/270111

I would recommend you shutdown fortilink ("set status down" in CLI) when you test VLAN connections on the Cisco side.

Toshi

[teksolutions · Accepted solution]

Downing the Fortilink interface didn't solve the problem but fully deleting all the Fortilink configuration did.... I guess that if you have Fortilink setup it screws up the trunks with other brands of switches. Super lame as this means you can't mix Fortinet switches with other brands. Just reenforces why you shouldn't ever buy FortiSwitches... 

[Toshi_Esumi]

Not because of lameness but because FGT managed FSWs over fortilink have special management scheme intending to make the management for average installation easier/more convenient at the FGT, instead of getting into indivitual FSWs. As the result, it adds some limitations and conditions to make the entire L2 network work as intended including the FGT.
If you don't have good grasp on those details, you might encounter L2 problem on the FGT, which affects to other L2 devices connected to the same FGT.

To avoid that, you always have an option not to use the fortilink management and put the FSWs on standalone mode, then mix them with any other switches, like Cisco's, which I prefer.

Toshi

[teksolutions]

Or you can buy better switches for better prices and not have to deal with the horrible FortiSwitch product altogether.... which I prefer.

[BSeklecki_GE]

I see your problem; encountered it 20 years ago when I first start commercializing FreeBSD/NetBSD Firewalls (which are still far superior to all this stuff, if you look closely; certainly anything made based on GNU/Linux):

Cisco native VLAN feature runs untagged, not tagged.

Normally on a 99.99% scenario, if you Wireshark capture to examine, the native VLAN (Default) of VLAN #1, is untagged on a trunk port.

Here, you've told the Cisco LACP/Switchport trunk to transmit VLAN#10 as untagged on that LACP Trunk. 

Simple misunderstanding that caught me up too:

So on the Fortinet side, you need to specify a the matching native/untagged ("Native") VLAN for the LACP LAG/Channel for your Layer3 interface.

I'm not even sure if the Fortinet can even do that; I've learned only recently how limited the IRB/CBR software switching functionality in the FortiOS software platform is on the FortiGate hardware.

(If you were running a native GNU/Linux firewall, this is is easy with BRCTL; but on Fortinet you'll have to figure out what FortiOS CLI Syntax translates to the correct BRCTL)

https://marc.info/?l=openbsd-tech&m=113471584916510&w=2
https://marc.info/?l=openbsd-misc&m=119298895920080&w=2