802.3ad Aggregation Link Is Up But Won't Respond

[rharms_tarc]

We have two WAN links from two different ISPs coming into our active/passive HA pair of FortiGate 300Es running v7.4.7 build2731 (Mature).

Currently there is no aggregation or load balancing in place. They are just two separate circuits, and one isn't really being utilized much day to day.

In an effort to improve that, I am working toward implementing SD-WAN on the FortiGates. To minimize downtime, I am setting up new links for the SD-WAN zone on unused ports.

We only have one port available from the ISP router at present, so have to run through a layer 2 switch in order to provide service to both FortiGates. The plan is to replace the single switch with a stacked pair to eliminate that switch as a single point of failure. While waiting for the stacked switches to be available, I've set things up as shown in the attached diagram so I can continue getting things prepared.

However, the 802.3ad agg link I created on the FortiGate doesn't seem to be working as it isn't pingable even from the layer 2 switch that is directly connected.

The config for the agg link on the FortiGate is:

config system interface
edit "LUMEN_ISP_AGG"
set vdom "root"
set ip 216.248.xxx.108 255.255.255.240
set allowaccess ping
set type aggregate
set member "port5" "port7"
set estimated-upstream-bandwidth 1000
set estimated-downstream-bandwidth 1000
set role wan
set snmp-index 82
set ip-managed-by-fortiipam disable
next
end

Running "diag netlink aggregate name LUMEN_ISP_AGG" on the FortiGate gives me the following.

HA1-300E (root) # diag netlink aggregate name LUMEN_ISP_AGG
LACP flags: (A|P)(S|F)(A|I)(I|O)(E|D)(E|D)
(A|P) - LACP mode is Active or Passive
(S|F) - LACP speed is Slow or Fast
(A|I) - Aggregatable or Individual
(I|O) - Port In sync or Out of sync
(E|D) - Frame collection is Enabled or Disabled
(E|D) - Frame distribution is Enabled or Disabled

status: up
npu: y
flush: n
asic helper: y
oid: 216
ports: 2
link-up-delay: 50ms
min-links: 1
ha: master
distribution algorithm: L4
LACP mode: active
LACP speed: slow
LACP HA: enable
aggregator ID: 4
actor key: 17
actor MAC address: e8:1c:ba:e5:a2:fc
partner key: 2
partner MAC address: f0:25:72:fd:91:00

member: port5
index: 0
link status: up
link failure count: 0
permanent MAC addr: e8:1c:ba:e5:a2:fc
LACP state: established
LACPDUs RX/TX: 1838/1683
actor state: ASAIEE
actor port number/key/priority: 1 17 255
partner state: ASAIEE
partner port number/key/priority: 262 2 32768
partner system: 32768 f0:25:72:fd:91:00
aggregator ID: 4
speed/duplex: 1000 1
RX state: CURRENT 6
MUX state: COLLECTING_DISTRIBUTING 4

member: port7
index: 1
link status: up
link failure count: 0
permanent MAC addr: e8:1c:ba:e5:a2:fe
LACP state: established
LACPDUs RX/TX: 1840/1683
actor state: ASAIEE
actor port number/key/priority: 2 17 255
partner state: ASAIEE
partner port number/key/priority: 263 2 32768
partner system: 32768 f0:25:72:fd:91:00
aggregator ID: 4
speed/duplex: 1000 1
RX state: CURRENT 6
MUX state: COLLECTING_DISTRIBUTING 4

and running a packet sniff shows LACPDUs

HA1-300E (root) # diag sniffer packet LUMEN_ISP_AGG "ether proto 0x8809" 6 0 a
interfaces=[LUMEN_ISP_AGG]
filters=[ether proto 0x8809]
2025-04-15 16:56:33.989204 LUMEN_ISP_AGG -- 802.3ad LACPDU (32768,F0-25-72-FD-91-00,0002,32768,0263) ASAIEE (65535,E8-1C-BA-E5-A2-FC,0017,0255,0002) ASAIEE
0x0000 0180 c200 0002 f025 72fd 9106 8809 0101 .......%r.......
0x0010 0114 8000 f025 72fd 9100 0002 8000 0107 .....%r.........
0x0020 3d00 0000 0214 ffff e81c bae5 a2fc 0011 =...............
0x0030 00ff 0002 3d00 0000 0310 8000 0000 0000 ....=...........
0x0040 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0050 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0060 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0070 0000 0000 0000 0000 0000 0000 ............

2025-04-15 16:56:46.586739 LUMEN_ISP_AGG -- 802.3ad LACPDU (32768,F0-25-72-FD-91-00,0002,32768,0262) ASAIEE (65535,E8-1C-BA-E5-A2-FC,0017,0255,0001) ASAIEE
0x0000 0180 c200 0002 f025 72fd 9105 8809 0101 .......%r.......
0x0010 0114 8000 f025 72fd 9100 0002 8000 0106 .....%r.........
0x0020 3d00 0000 0214 ffff e81c bae5 a2fc 0011 =...............
0x0030 00ff 0001 3d00 0000 0310 8000 0000 0000 ....=...........
0x0040 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0050 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0060 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0070 0000 0000 0000 0000 0000 0000 ............

2025-04-15 16:57:01.750867 LUMEN_ISP_AGG -- 802.3ad LACPDU (32768,F0-25-72-FD-91-00,0002,32768,0263) ASAIEE (65535,E8-1C-BA-E5-A2-FC,0017,0255,0002) ASAIEE
0x0000 0180 c200 0002 f025 72fd 9106 8809 0101 .......%r.......
0x0010 0114 8000 f025 72fd 9100 0002 8000 0107 .....%r.........
0x0020 3d00 0000 0214 ffff e81c bae5 a2fc 0011 =...............
0x0030 00ff 0002 3d00 0000 0310 8000 0000 0000 ....=...........
0x0040 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0050 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0060 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0070 0000 0000 0000 0000 0000 0000 ............

but that's as far as I'm getting at this point.

[Dan_Eng52]

Hi rharms_tarc.

I hope you're well. 

Can you try adding the below command to your LAG interface: 

set lacp-mode static

Often when I have a LAG created that is connected to another vendor switch i.e. Aruba, Cisco etc this command is required in order to bring the LAG interface up. I couldn't see this command added in the output.

Please give that a try and let me know how it goes. 

Thanks, 

Dan. 

[rharms_tarc]

I tried changing the lacp mode setting to static.  It caused all of the port-channel ports on the Cisco switch to drop from status bndl to status indep.  I had to change the port-channel configs from mode "active" to mode "on" to get the port-channels to come back up.  Still couldn't ping the IP of the aggregated interface on the FortiGate with everything in static mode.  Switched all ports back on Cisco and FortiGate back to active mode and the LACP bundles immediately re-formed.  

[Toshi_Esumi]

At least one side has to be "Active", while the other side can be "on/static/passive". Only active side initiates the negotiation/handshake. In other words, both sides can be active.
If it's up, as you showed, don't change that part of setting. That's not the problem in your situation.

Toshi

[Toshi_Esumi]

You showed LAG/LACP is perfectly fine on the FGT side. Just make sure the Catalyst sees the same with "show lacp" commands.

I think the problem is Catalyst's switching to connect L2 between the FGT's LAG port to the Cisco 2901 port. Since you configured the IP on the LAG itself, it's untagged interface. Check both sides on the switch is on the same VLAN (likely VLAN1). Then hook up a laptop or something to another port and assign the same or another IP in the /28 subnet, then test toward both ends. Also you can set up a mirror port to sniff between two ends. But I don't think it's not necessary. Likely a simple misconfig at the switch.

Toshi

[rharms_tarc]

I verified that the Catalyst also sees everything with the port channels as being good:

ISP_2960G_SW1#show lacp internal
Flags: S - Device is requesting Slow LACPDUs
F - Device is requesting Fast LACPDUs
A - Device is in Active mode P - Device is in Passive mode

Channel group 1
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Gi0/3 SA bndl 32768 0x1 0x1 0x104 0x3D
Gi0/4 SA bndl 32768 0x1 0x1 0x105 0x3D

Channel group 2
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Gi0/5 SA bndl 32768 0x2 0x2 0x106 0x3D
Gi0/6 SA bndl 32768 0x2 0x2 0x107 0x3D

The 2901 shouldn't enter into it at all.  I'm trying to ping from the Cisco switch command line to the FortiGate, so that's all inside of the 2901.  All Cisco switch ports involved are set as access ports assigned to the same VLAN.  I can ping from the Cisco switch to the pre-existing non-aggregated FortiGate interface, so it should seemingly also work to ping to the aggregated interface, it just doesn't.  

[Toshi_Esumi]

Where did you configured the Catalyst's IP? Under "interface vlan 1"?
Share us the port-channel config.

Toshi

[rharms_tarc]

spanning-tree mode rapid-pvst
spanning-tree loopguard default
spanning-tree portfast default
spanning-tree portfast bpduguard default
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
vlan 106
name MGMT_VLAN
!
vlan 2001
name ISP1_LUMEN_NO*IP
!
vlan 2002
name ISP2_SPECTRUM_NO*IP

!
interface Port-channel1
switchport access vlan 2001
!
interface Port-channel2
switchport access vlan 2001
!
interface GigabitEthernet0/1
description LUMEN_ISP_0/0/2
switchport access vlan 2001
flowcontrol receive on
!
interface GigabitEthernet0/2
description TEMPORARY_OLD_LUMEN
switchport access vlan 2001
flowcontrol receive on
!
interface GigabitEthernet0/3
description FORTIGATE1_PORT5
switchport access vlan 2001
channel-group 1 mode active
!
interface GigabitEthernet0/4
description TEMP FG1_PORT7
switchport access vlan 2001
channel-group 1 mode active
!
interface GigabitEthernet0/5
description FORTIGATE2_PORT5
switchport access vlan 2001
channel-group 2 mode active
!
interface GigabitEthernet0/6
description TEMP FG2_PORT7
switchport access vlan 2001
channel-group 2 mode active
!
interface GigabitEthernet0/7
description CR_2921_VG1_SIP
switchport access vlan 2001
flowcontrol receive on
!
interface GigabitEthernet0/8
switchport access vlan 2001
flowcontrol receive on
!
interface GigabitEthernet0/9
description CR_2921_VG2_SIP
switchport access vlan 2001
flowcontrol receive on
!
interface GigabitEthernet0/10
description TEMPORARY_FG1P15
switchport access vlan 2001
flowcontrol receive on
!
interface GigabitEthernet0/11
description SPECTRUM_ISP_0/1
switchport access vlan 2002
flowcontrol receive on
!
interface GigabitEthernet0/12
switchport access vlan 2002
flowcontrol receive on
!
interface GigabitEthernet0/13
description FORTIGATE1_PORT6
switchport access vlan 2002
shutdown
!
interface GigabitEthernet0/14
switchport access vlan 2002
!
interface GigabitEthernet0/15
description FORTIGATE2_PORT6
switchport access vlan 2002
shutdown
!
interface GigabitEthernet0/16
switchport access vlan 2002
!
interface GigabitEthernet0/17
switchport access vlan 2002
flowcontrol receive on
!
interface GigabitEthernet0/18
switchport access vlan 2002
flowcontrol receive on
!
interface GigabitEthernet0/19
switchport access vlan 2002
flowcontrol receive on
!
interface GigabitEthernet0/20
description TEMPORARY_FG1P16
switchport access vlan 2002
srr-queue bandwidth share 1 30 35 5
priority-queue out
mls qos trust cos
flowcontrol receive on
auto qos trust
spanning-tree bpduguard disable
!
interface GigabitEthernet0/21
switchport access vlan 106
flowcontrol receive on
!
interface GigabitEthernet0/22
switchport access vlan 106
flowcontrol receive on
!
interface GigabitEthernet0/23
description plug_in_yo_laptop
switchport access vlan 106
flowcontrol receive on
!
interface GigabitEthernet0/24
description Uplink Port
switchport trunk native vlan 106
switchport trunk allowed vlan 106
switchport mode trunk
srr-queue bandwidth share 1 30 35 5
priority-queue out
mls qos trust cos
auto qos trust
spanning-tree bpduguard disable
!
interface Vlan1
no ip address
no ip redirects
no ip proxy-arp
shutdown
!
interface Vlan106
ip address 10.245.xxx.24 255.255.254.0
no ip redirects
no ip unreachables
no ip proxy-arp
!

The only IP on the Cisco switch is configured on our management VLAN. The VLAN that I'm trying to troubleshoot currently has no IP address anyplace. 

[Toshi_Esumi]

You wouldn't be able to ping IP on VLAN 2001 if you don't have IP configured under "interface Vlan2001" in the same /28 subnet on the L2 switch.

[rharms_tarc]

Right, but I'm not trying to ping anything on VLAN 2001.  I'm trying to ping the aggregated FortiGate interface that is attached to channel-group 1 on the Cisco switch.  From the CLI on the switch, I can ping the non-aggregated FortiGate interface that is attached to Gi0/10 (part of VLAN 2001) on the switch.  I can also ping from the switch CLI to the 2901 gateway router that is attached to Gi0/2 (part of VLAN 2001) on the switch.  So ping from the switch to other devices works, it's just this aggregated interface that doesn't.