Technical Tip: Initial troubleshooting steps for LACP (Link Aggregation - 802.3ad)
Description
This article provides troubleshooting commands that can be used when facing LACP (Link Aggregation Control Protocol) issues on a FortiGate. The related articles provide additional information about LACP.
Scope
Solution
There are three modes of LACP on the FortiGate:
- Active: Actively use LACP to negotiate 802.3ad link aggregation by initiating the negotiation through LACP packet exchanges.
- Passive: Passively use LACP to negotiate 802.3ad link aggregation by responding to LACP packets without initiating the negotiation.
- Static: use static aggregation, do not send and ignore any LACP messages (all ports in the LAG will send traffic).
Depending on the remote device, it may be necessary to adapt the LACP mode appropriately.
The 'lacp-ha-secondary enable' command allows subordinate units in an HA Cluster to participate in LACP negotiation by allowing them to send/receive LACP messages. When disabled, it blocks HA secondary units from sending/receiving LACP messages.
The lacp-speed determines how often the interface sends LACP messages. By default, it is set to slow, which sends LACP messages every 30 seconds. However, when it is set too fast, it sends an LACP message every second.
There are three types of traffic distribution across the ports in the LACP bundle. Distribution of sessions uses a hash of either L2/L3/L4 header fields divided by the number of physical interfaces in the link aggregation group to determine a remainder value that identifies the link number to use.
Example of an LACP configuration:
config system interface
edit "lacp_ports"
set vdom "root"
set type aggregate
set member "port1" "port2"
set description "lacp_example"
set lacp-mode active <----- Default.
set lacp-ha-secondary enable <----- Default.
set lacp-speed slow <----- Default.
set algorithm L4 <----- Default.
next
end
The following CLI commands can be used to check the ports and LAG (Link Aggregation Group) status.
- Example of LACP operational information when ports are up and in the LAG.
diagnose netlink aggregate name the_aggregate_link
LACP flags: (A|P)(S|F)(A|I)(I|O)(E|D)(E|D)
(A|P) - LACP mode is Active or Passive
(S|F) - LACP speed is Slow or Fast
(A|I) - Aggregatable or Individual
(I|O) - Port In sync or Out of sync
(E|D) - Frame collection is Enabled or Disabled
(E|D) - Frame distribution is Enabled or Disabled
status: up
npu: n
flush: n
asic helper: y
ports: 1
link-up-delay: 50ms
min-links: 1
ha: master
distribution algorithm: L4
LACP mode: active
LACP speed: slow
LACP HA: enable
aggregator ID: 1
ports: 2
actor key: 17
actor MAC address: 00:09:0f:68:35:94
partner key: 17
partner MAC address: 00:09:0f:68:37:d8
member: port7
status: up
link failure count: 3
permanent MAC addr: 00:09:0f:68:35:94
actor state: ASAIEE
partner state: ASAIEE
aggregator ID: 1
member: port8
status: up
link failure count: 2
permanent MAC addr: 00:09:0f:68:35:95
actor state: ASAIEE
partner state: ASAIEE
aggregator ID: 1
In this example, the aggregator IDs have the same value on both ports and globally (ID=1). This means that both ports are operational in the LAG.
- Example of LACP operational information when both ports are up, but there is no LACPDU exchange on port 5(*).
diagnose netlink aggregate name the_aggregate_link
LACP flags: (A|P)(S|F)(A|I)(I|O)(E|D)(E|D)
(A|P) - LACP mode is Active or Passive
(S|F) - LACP speed is Slow or Fast
(A|I) - Aggregatable or Individual
(I|O) - Port In sync or Out of sync
(E|D) - Frame collection is Enabled or Disabled
(E|D) - Frame distribution is Enabled or Disabled
status: up
distribution algorithm: L3
LACP mode: active
LACP speed: slow
LACP HA: enable
aggregator ID: 1
ports: 1
actor key: 17
actor MAC address: 00:09:0f:71:1f:22
partner key: 45
partner MAC address: 00:0d:66:2f:2b:40
secondary: port5
status: up
link failure count: 19
permanent MAC addr: 00:09:0f:71:1f:22
actor state: ASAIDD <----- Disabled.
partner state: ASIODD <----- Out of sync/Disabled.
aggregator ID: 2
secondary: port6
status: up
link failure count: 2
permanent MAC addr: 00:09:0f:71:1f:23
actor state: ASAIEE
partner state: ASAIEE
aggregator ID: 1
Note that port5 is in a different aggregator (2) above than the global aggregator ID (1). In this case, only port6 is operational in the LAG.
(*) If both FortiGates or the equipment are connected via an intermediate L2 switch, make sure that it passes LACPDU packets.
Link Aggregation Control Protocol "LACPDU" packet format and how to get a sniffer trace from the CLI:
diagnose sniffer packet the_aggregate_link
2.546898 aggreg_link -- 802.3ad LACPDU (65535,00-09-0F-68-37-D8,0017,0255,0002) ASAIEE (65535,00-09-0F-68-35-94,0017,0255,0002) ASAIEE
0x0000 0180 c200 0002 0009 0f68 37d9 8809 0101 .........h7......
Dst Multicast - src=lowest MAC of all ports in the LAG - Eth frame type.
If there is a lot of traffic, only LACP traffic can be fully captured with:
diagnose sniffer packet any "ether proto 0X8809" 6 0 a
The sniffer will run indefinitely until it is stopped. To stop the sniffer, press Ctrl + C.
- Example of LACP configurations being identical on both peers, however SYNC flag still shows discrepancy:
diagnose netlink aggregate name the_aggregate_link
member: x2
index: 0
link status: up
link failure count: 0
permanent MAC addr: 44:33:22:11:bb:55
LACP state: established
LACPDUs RX/TX: 69/61
actor state: ASAIEE
actor port number/key/priority: 2 33 255
partner state: ASAIEE
partner port number/key/priority: 65 100 32768
partner system: 4096 aa:bb:cc:dd:ee:ff
aggregator ID: 2
speed/duplex: 10000 1
RX state: CURRENT 6
MUX state: COLLECTING_DISTRIBUTING 4
member: x1
index: 1
link status: up
link failure count: 0
permanent MAC addr: 44:33:22:11:bb:56
LACP state: negotiating
LACPDUs RX/TX: 6/3
actor state: ASAODD ----------------------> Out Of Sync
actor port number/key/priority: 3 33 255
partner state: ASAIDD --------------------> In Sync
partner port number/key/priority: 33 101 32768
partner system: 4096 aa:bb:cc:dd:ee:ff
aggregator ID: 1
speed/duplex: 10000 1
RX state: CURRENT 6
MUX state: WAITING 2
The Actor/Partner Key are unique identifiers combining System Priority (configurable), MAC Address, PortID, priorities, speed, duplex..etc. These are some critical parameters in LACP used in key generation.
Each peer (partner) should have same partner key for all members of the LACP bundle. In the example above both FortiGate interfaces have same actor key 33, however the partner key for x1 is 101, which does not reflect with the partner key of member x2 (i.e 100) which is already in sync. This will hint that x2 is not communicating with the same aggregate configuration as x1 on the other side.In most cases, this could happen if physical wiring does not align with the actual configurations either on the FortiGate or the switch.
The following information should be provided when opening a ticket with TAC Support for an LACP issue:
- The FortiGate configuration file.
- Information about how the two devices are connected for this LACP bundle (direct cables or fibers/Intermediate L2 or metro device between the FortiGate and the other device).
- Results of the following CLI commands:
diagnose netlink aggregate name the_aggregate_link
diagnose hardware deviceinfo nic <all_interfaces_in_the_aggregation>
diagnose sniffer packet the_aggregate_link " " 6 0 l
diagnose sniffer packet any "ether proto 0X8809" 6 0 a
- The output of the sniffer trace is gathered on the other end (port-mirroring or PCAP).
- For a single session, LACP can distribute traffic across multiple interfaces, which may lead to packet reordering issues for applications that require ordered packet delivery. If this is causing problems, consider using static aggregation, aligning hash settings, or temporarily disabling one interface to ensure that all traffic for a session is handled consistently.
Related articles:
Technical Tip / FAQ: FortiGate and FortiOS support for 802.3ad (LACP - Link Aggregation)
Troubleshooting Tip: LACP issue
Technical Tip: Link aggregation (IEEE 802.3ad) Interface Flapping When Adding or Removing Members