Technical Tip: Initial troubleshooting steps for LACP (Link Aggregation - 802.3ad)

Description

This article describes the troubleshooting commands for analyzing LACP (Link Aggregation Control Protocol) issues on a FortiGate. Refer to the related articles for further details about LACP.

Scope

FortiGate.

Solution

There are three modes of LACP on the FortiGate:

• Active: Actively use LACP to negotiate 802.3ad link aggregation by initiating the negotiation through LACP packet exchanges.

• Passive: Passively use LACP to negotiate 802.3ad link aggregation by responding to LACP packets without initiating the negotiation.

• Static: Use static aggregation, do not send and ignore any LACP messages (all ports in the LAG will send traffic).

Depending on the remote device, it may be necessary to adapt the LACP mode appropriately.

The 'lacp-ha-secondary enable' command allows subordinate units in an HA Cluster to participate in LACP negotiation by allowing them to send/receive LACP messages. When disabled, it blocks HA secondary units from sending/receiving LACP messages.

The lacp-speed determines how often the interface sends LACP messages. By default, it is set to slow, which sends LACP messages every 30 seconds. However, when it is set too fast, it sends an LACP message every second.

There are three types of traffic distribution across the ports in the LACP bundle. Distribution of sessions uses a hash of either L2/L3/L4 header fields divided by the number of physical interfaces in the link aggregation group to determine a remainder value that identifies the link number to use.

Example of an LACP configuration:

• The last four 'set' parameters are the default values.

config system interface
    edit "lacp_ports"
        set vdom "root"
        set type aggregate
        set member "port1" "port2"
        set description "lacp_example"
        set lacp-mode active            
        set lacp-ha-secondary enable    
        set lacp-speed slow             
        set algorithm L4
    next
end

The following CLI commands can be used to check the ports and LAG (Link Aggregation Group) status.

1. Example of LACP operational information when ports are up and in the LAG.

diagnose netlink aggregate name the_aggregate_link

LACP flags: (A|P)(S|F)(A|I)(I|O)(E|D)(E|D)
(A|P) - LACP mode is Active or Passive
(S|F) - LACP speed is Slow or Fast
(A|I) - Aggregatable or Individual
(I|O) - Port In sync or Out of sync
(E|D) - Frame collection is Enabled or Disabled
(E|D) - Frame distribution is Enabled or Disabled

status: up
npu: n
flush: n
asic helper: y
ports: 1
link-up-delay: 50ms
min-links: 1
ha: master
distribution algorithm: L4
LACP mode: active
LACP speed: slow
LACP HA: enable
aggregator ID: 1
ports: 2
actor key: 17
actor MAC address: 00:09:0f:68:35:94
partner key: 17
partner MAC address: 00:09:0f:68:37:d8

member: port7
status: up
link failure count: 3
permanent MAC addr: 00:09:0f:68:35:94
actor state: ASAIEE
partner state: ASAIEE
aggregator ID: 1

member: port8
status: up
link failure count: 2
permanent MAC addr: 00:09:0f:68:35:95
actor state: ASAIEE
partner state: ASAIEE
aggregator ID: 1

In this example, the aggregator IDs have the same value on both ports and globally (ID=1). This means that both ports are operational in the LAG.

2. Example of LACP operational information when both ports are up, but there is no LACPDU exchange on port 5(*).

diagnose netlink aggregate name the_aggregate_link

LACP flags: (A|P)(S|F)(A|I)(I|O)(E|D)(E|D)
(A|P) - LACP mode is Active or Passive
(S|F) - LACP speed is Slow or Fast
(A|I) - Aggregatable or Individual
(I|O) - Port In sync or Out of sync
(E|D) - Frame collection is Enabled or Disabled
(E|D) - Frame distribution is Enabled or Disabled

status: up
distribution algorithm: L3
LACP mode: active
LACP speed: slow
LACP HA: enable
aggregator ID: 1
ports: 1
actor key: 17
actor MAC address: 00:09:0f:71:1f:22
partner key: 45
partner MAC address: 00:0d:66:2f:2b:40

secondary: port5
status: up
link failure count: 19
permanent MAC addr: 00:09:0f:71:1f:22
actor state: ASAIDD                     <----- Disabled.
partner state: ASIODD                   <----- Out of sync/Disabled.
aggregator ID: 2

secondary: port6
status: up
link failure count: 2
permanent MAC addr: 00:09:0f:71:1f:23
actor state: ASAIEE
partner state: ASAIEE
aggregator ID: 1

In the above:

• 'actor state: ASAIDD' means it is disabled.

• 'partner state: ASIODD' means it is out of sync/disabled.

Note: port5 is in a different aggregator (2) above than the global aggregator ID (1). In this case, only port6 is operational in the LAG.

(*) If both FortiGates or the equipment are connected via an intermediate L2 switch, make sure that it passes LACPDU packets.

Link Aggregation Control Protocol "LACPDU" packet format and how to get a sniffer trace from the CLI:

diagnose sniffer packet the_aggregate_link

2.546898 aggreg_link -- 802.3ad LACPDU (65535,00-09-0F-68-37-D8,0017,0255,0002) ASAIEE (65535,00-09-0F-68-35-94,0017,0255,0002) ASAIEE
0x0000 0180 c200 0002 0009 0f68 37d9 8809 0101 .........h7......

Destination MAC- Source MAC(lowest MAC of all ports in the LAG) - Eth frame type.

If there is a lot of traffic, only LACP traffic can be fully captured with:

diagnose sniffer packet any "ether proto 0X8809" 6 0 a

The sniffer will run indefinitely until it is stopped. To stop the sniffer, press Ctrl + C.

3. Example of LACP configurations being identical on both peers, however SYNC flag still shows discrepancy:

diagnose netlink aggregate name the_aggregate_link

member: x2
index: 0
link status: up
link failure count: 0
permanent MAC addr: 44:33:22:11:bb:55
LACP state: established
LACPDUs RX/TX: 69/61
actor state: ASAIEE
actor port number/key/priority: 2 33 255
partner state: ASAIEE
partner port number/key/priority: 65 100 32768
partner system: 4096 aa:bb:cc:dd:ee:ff
aggregator ID: 2
speed/duplex: 10000 1
RX state: CURRENT 6
MUX state: COLLECTING_DISTRIBUTING 4

member: x1
index: 1
link status: up
link failure count: 0
permanent MAC addr: 44:33:22:11:bb:56
LACP state: negotiating
LACPDUs RX/TX: 6/3
actor state: ASAODD ----------------------> Out Of Sync
actor port number/key/priority: 3 33 255
partner state: ASAIDD --------------------> In Sync
partner port number/key/priority: 33 101 32768
partner system: 4096 aa:bb:cc:dd:ee:ff
aggregator ID: 1
speed/duplex: 10000 1
RX state: CURRENT 6
MUX state: WAITING 2

4. Example of LACP output when LACP member is stuck in LACP_Disabled state

Under very specific timing conditions during port recovery, an LACP member port may not receive the correct speed and duplex settings. As a result, the port can become stuck in the LACP_DISABLED state and cease LACP negotiation.

In the following:

• 'actor port number/key/priority: 1 0 255' is the LACP actor port key showing the '0' value.
  
  

diagnose netlink aggregate name the_aggregate_link

member: port1
  index: 0
  link status: up
  link failure count: 0
  permanent MAC addr: e8:ed:d6:09:d7:b8
  LACP state: negotiating
  actor state: ASAODD 
  actor port number/key/priority: 1 0 255
  partner state: ASIODD
  partner port number/key/priority: 1 1 255
  partner system: 64696 00:00:00:00:00:00
  aggregator ID: 2 
  speed/duplex: 1000 1
  RX state: LACP_DISABLED 3
  MUX state: WAITING 2

Workaround: Bounce the port to re-negotiate LACP.

The Actor/Partner Key are unique identifiers combining System Priority (configurable), MAC Address, PortID, priorities, speed, duplex..etc. These are some critical parameters in LACP used in key generation.

Each peer (partner) should have same partner key for all members of the LACP bundle. In the example above both FortiGate interfaces have same actor key 33, however the partner key for x1 is 101, which does not reflect with the partner key of member x2 (i.e 100) which is already in sync. This will hint that x2 is not communicating with the same aggregate configuration as x1 on the other side.In most cases, this could happen if physical wiring does not align with the actual configurations either on the FortiGate or the switch.

The following information should be provided when opening a ticket with TAC Support for an LACP issue:

• The FortiGate configuration file.

• Information about how the two devices are connected for this LACP bundle (direct cables or fibers/Intermediate L2 or metro device between the FortiGate and the other device).

• Results of the following CLI commands:

diagnose netlink aggregate name the_aggregate_link
diagnose hardware deviceinfo nic <all_interfaces_in_the_aggregation>
diagnose sniffer packet the_aggregate_link " " 6 0 l
diagnose sniffer packet any "ether proto 0X8809" 6 0 a

• The output of the sniffer trace is gathered on the other end (port-mirroring or PCAP).

• For a single session, LACP can distribute traffic across multiple interfaces, which may lead to packet reordering issues for applications that require ordered packet delivery. If this is causing problems, consider using static aggregation, aligning hash settings, or temporarily disabling one interface to ensure that all traffic for a session is handled consistently.

Related articles:

• Technical Tip: FortiGate HA A-P (Active-Passive) cluster connected to a L2 switch with LACP (802.3ad)

• Technical Tip: Understanding outputs of LACP related debug commands and what parameters need to match in LACP neighbors to make the link UP

• Technical Tip: Link aggregation Control Protocol (LACP) Design and Topology

• Technical Tip: Link aggregation limitation and maximum supported number of interfaces

• Technical Tip: Link aggregation (IEEE 802.3ad) Interface Flapping When Adding or Removing Members