Technical Tip: Configuring BGP TCP Authentication Option (TCP-AO) Between FortiGate and Third party vendor

Background

TCP Authentication Option (TCP-AO) enhances BGP session security by replacing TCP-MD5 with modern, stronger cryptographic methods.
FortiOS supports TCP-AO starting with version 7.4.2 and uses key chains with HMAC-SHA1, HMAC-SHA256, or CMAC-AES128 algorithms.

However, interoperability between vendors requires precise alignment of:

• Cryptographic algorithm.
• Key-chain configuration.
• TCP option inclusion.
• Key lifetimes.
• AO MAC behavior.

Any mismatch causes the peer to silently drop SYN packets, preventing BGP from forming.

FortiGate Configuration:

Creating a Key-Chain for TCP-AO (FortiGate):

Example using HMAC-SHA256:

config router key-chain
    edit "BGP-AO-CHAIN"
        config key
            edit "1"
                set accept-lifetime 00:00:00 01 01 2024 2147483646
                set send-lifetime 00:00:00 01 01 2024 2147483646
                set key-string SuperSecretKey123
                set algorithm hmac-sha256
            next
        end
    next

end

Example using AES-128 CMAC:

config router key-chain
    edit "AES-CMAC-CHAIN"
        config key
            edit "1"
                set accept-lifetime 01:01:01 01 01 2021 2147483646
                set send-lifetime 01:01:01 01 01 2021 2147483646
                set key-string **********
                set algorithm cmac-aes128
            next
        end
    next
end

Applying the Key-Chain to a BGP Neighbor:

One command links the key-chain to the BGP Neighbor:

config router bgp
    set as 65000
        config neighbor
            edit "192.0.2.1"
                set auth-options "BGP-AO-CHAIN"
            next
        end
    end

FortiGate automatically assigns send/receive identifiers internally - there is no manual ID configuration on FortiOS.

Verifying TCP-AO on a FortiGate:

diagnose sys tcp-auth-options

The command confirms:

• The selected algorithm (AES128 / SHA1 / SHA256).
• The active key.
• The lifetime values.
• The peer address.

Cisco Configuration:

Key-Chain for TCP-AO:

key chain BGP-AO-CHAIN tcp
key 1
cryptographic-algorithm hmac-sha-256
key-string 7 SuperSecretKey123
send-id 10
recv-id 10
accept-lifetime 00:00:00 Jan 1 2025 infinite
send-lifetime 00:00:00 Jan 1 2025 infinite

Bind the key-chain to a BGP Neighbor.

router bgp 65000
address-family ipv4 vrf FW_TRAN
neighbor 192.0.2.254 remote-as 65000
neighbor 192.0.2.254 ao BGP-AO-CHAIN
neighbor 192.0.2.254 fall-over bfd single-hop
neighbor 192.0.2.254 activate
exit-address-family

Ensure TCP Options are included in AO MAC

Some Cisco platforms do not include MSS, SACK, timestamps, or window-scaling in the AO digest unless explicitly configured.

This will cause errors such as:

• AO invalid current key.
• bad seg - AO invalid current key.
• Invalid current key – aborting key creation.

Enforce option inclusion:

tcp ao include-options

This is required on multiple Cisco platforms for interoperability with FortiGate.

Juniper Configuration:

set routing-options tcp-session authentication algorithm hmac-sha-256
set routing-options tcp-session authentication key-chain BGP-AO-CHAIN key 1 key "SuperSecretKey123"
set routing-options tcp-session authentication send-id 10
set routing-options tcp-session authentication recv-id 10

set protocols bgp group FGT type internal
set protocols bgp group FGT local-address 192.0.2.1
set protocols bgp group FGT neighbor 192.0.2.254 peer-as 65000
set protocols bgp group FGT neighbor 192.0.2.254 tcp-session authentication key-chain BGP-AO-CHAIN

Troubleshooting: why TCP-AO BGP sessions fail to establish.

Peer silently drops SYN packets.

Symptoms:

• FortiGate repeatedly sends SYN with AO.
• No SYN-ACK or RST received.
• BGP stays in Connect / Idle.

Cause:
AO MAC mismatch -> peer discards SYN.

Algorithm compatibility problems:

Common platform behavior:

• Some routers support only HMAC-SHA1.
• Some support HMAC-SHA1 + SHA-256.
• Some do not support SHA-256 and silently drop packets.
• Some platforms accept AES-128 CMAC only in specific software versions.

Resolution:
If in doubt, configure HMAC-SHA1 on all devices.

TCP Options not included in AO Computation:

If one peer includes TCP:

• MSS.
• Timestamps.
• SACK.
• Window scaling.

If the other peer does not, AO digests differ.

Symptoms:

• AO invalid key.
• Invalid MAC.
• Idle/Connect looping.
• No SYN-ACK.

Fix:

tcp ao include-options

Key-Chain Not Activated / TCB Not Initialized:

Symptoms:

• tcb is not available.
• Active open failed.

Cause:

Key-chain not bound to AO subsystem:

Resolution:

• Recreate key-chain
• Reapply to BGP
• Verify TCB is active afterwards

Key-Chain lifetime or ID mismatches:

AO requires:

• Overlapping lifetimes.
• Matching IDs on non-FortiGate devices.
• Same algorithm.
• Same key-string.

Any mismatch prevents TCP session establishment.

Interoperability best practices:

• Verify algorithm support across all platforms.
• When uncertain, use HMAC-SHA1.
• Ensure overlapping key lifetimes.
• Match key strings exactly.
• Ensure IDs match on Cisco/Juniper.

For Cisco, always evaluate:

tcp ao include-options

Verify SYN packets contain the same TCP options.

Conclusion:

TCP-AO significantly enhances BGP security, but successful operation in multi-vendor environments requires strict alignment of algorithms, key parameters, and TCP option handling.