Technical Tip: Initial troubleshooting steps for LACP (Link Aggregation - 802.3ad)

[alif]

Description
This article provides troubleshooting commands that can be used when facing LACP (Link Aggregation Control Protocol) issues on a FortiGate.

The related articles provide additional information about LACP.

Scope

FortiGate supporting LACP: Models 310B (Recommended on port handled by the same NP2), 300A, 400A, 500A, and 800 or higher.

Solution
There are three modes of LACP on the FortiGate:

- Active: actively use LACP to negotiate 802.3ad aggregation.
- Passive: passively use LACP to negotiate 802.3ad aggregation.
- Static: use static aggregation, do not send and ignore any LACP messages (all ports in the LAG will send traffic).

Depending on the remote device, you might have to adapt the LACP mode appropriately.

The 'lacp-ha-slave enable' command allows subordinate units in HA Cluster to participate in LACP negotiation.

The lacp-speed determines how often the interface sends LACP messages. By default, it is set to slow which sends LACP messages every 30 seconds.

There are three types of traffic distribution across the ports in the LACP bundle. Distribution of sessions uses a hash of either L2 / L3 / L4 header fields divided by the number of physical interfaces in the link aggregation group to determine a remainder value that identifies the link number to use.

Example of an LACP configuration.

# config system interface
    edit "lacp_ports"
        set vdom "root"
        set type aggregate
        set member "port1" "port2"
        set description 'lacp_example'
        set lacp-mode active            <----- Default.
        set lacp-ha-slave enable        <----- Default.
        set lacp-speed slow             <----- Default.
        set algorithm L4                <----- Default.
    next
end

The following CLI commands can be used to check the ports and LAG (Link Aggregation Group) status.

1) Example of LACP operational information when ports are up and in the LAG.

FGT# diag netlink aggregate name your_aggregate_link

LACP flags: (A|P)(S|F)(A|I)(I|O)(E|D)(E|D)
(A|P) - LACP mode is Active or Passive
(S|F) - LACP speed is Slow or Fast
(A|I) - Aggregatable or Individual
(I|O) - Port In sync or Out of sync
(E|D) - Frame collection is Enabled or Disabled
(E|D) - Frame distribution is Enabled or Disabled

status: up
distribution algorithm: L4
LACP mode: active
LACP speed: slow
LACP HA: enable
aggregator ID: 1
ports: 2
actor key: 17
actor MAC address: 00:09:0f:68:35:94
partner key: 17
partner MAC address: 00:09:0f:68:37:d8
slave: port7
status: up
link failure count: 3
permanent MAC addr: 00:09:0f:68:35:94
actor state: ASAIEE
partner state: ASAIEE
aggregator ID: 1

slave: port8
status: up
link failure count: 2
permanent MAC addr: 00:09:0f:68:35:95
actor state: ASAIEE
partner state: ASAIEE
aggregator ID: 1

In this example the aggregator IDs have the same value on both ports and globally (ID=1), this means that both ports are operational in the LAG.

2) Example of LACP operational information when both ports are up, but there is no LACPDU exchange on port 5(*).

FGT# diag netlink aggregate name your_aggregate_link

LACP flags: (A|P)(S|F)(A|I)(I|O)(E|D)(E|D)
(A|P) - LACP mode is Active or Passive
(S|F) - LACP speed is Slow or Fast
(A|I) - Aggregatable or Individual
(I|O) - Port In sync or Out of sync
(E|D) - Frame collection is Enabled or Disabled
(E|D) - Frame distribution is Enabled or Disabled

status: up
distribution algorithm: L3
LACP mode: active
LACP speed: slow
LACP HA: enable
aggregator ID: 1
ports: 1
actor key: 17
actor MAC address: 00:09:0f:71:1f:22
partner key: 45
partner MAC address: 00:0d:66:2f:2b:40

slave: port5
status: up
link failure count: 19
permanent MAC addr: 00:09:0f:71:1f:22
actor state: ASAIDD                     <----- DISABLED.
partner state: ASIODD                   <----- OUT OF SYNC / DISABLED.
aggregator ID: 2

slave: port6
status: up
link failure count: 2
permanent MAC addr: 00:09:0f:71:1f:23
actor state: ASAIEE
partner state: ASAIEE
aggregator ID: 1

See from above that port5 is in a different aggregator (2) then the global aggregator ID (1). In this case, only port6 is operational in the LAG.

(*) If both FortiGates or equipment are connected via an intermediate L2 switch, make sure that is passes LACPDU packets.

Link Aggregation Control Protocol "LACPDU" packet format and how to get a sniffer trace from the CLI:

FGT # diag sniffer packet your_aggregate_link

2.546898 aggreg_link -- 802.3ad LACPDU (65535,00-09-0F-68-37-D8,0017,0255,0002) ASAIEE (65535,00-09-0F-68-35-94,0017,0255,0002) ASAIEE
0x0000 0180 c200 0002 0009 0f68 37d9 8809 0101 .........h7......

Dst Multicast - Src = lowest MAC of all ports in the LAG - Eth frame type

If there is a lot of traffic, only LACP traffic can be fully captured with:

# diagnose sniffer packet any "ether proto 0X8809" 6 0 a

The following information should be provided when opening a ticket with TAC Support for an LACP issue:
- Configuration file of the FortiGate.
- How are the two devices connected together for this LACP bundle (direct cables or fibers/Intermediate L2 or metro device between the FortiGate and the other device).
- Result of the following CLI commands:

# diag netlink aggregate name your_aggregate_link
# diagnose hardware deviceinfo nic <all_interface_in_your_aggregation>
# diag sniff packet your_aggregate_link " " 6 0 l

- Output of the sniffer trace gathered on the other end (port-mirroring or PCAP).

Related Articles

FortiGate-310B and FortiGate-620B LACP (802.3ad aggregate port) configuration

Technical Note / FAQ: FortiGate and FortiOS support for 802.3ad (LACP - Link Aggregation)

Technical Note : FortiGate HA A-P (Active-Passive) cluster connected to a L2 switch with LACP (802.3...