Technical Tip: Configuring BGP TCP Authentication Option (TCP-AO) Between FortiGate and Third party vendor

[amalsky]

Description	This article describes how to configure TCP Authentication Option (TCP-AO) for BGP peering between FortiGate firewalls and third-party vendors (Cisco or Juniper routers, for example). It also provides comprehensive troubleshooting guidance based on real interoperability issues, including: AO key validation failures Silent SYN drops Algorithm incompatibilities Missing TCP options in AO calculation TCP Control Block (TCB) is not initialising Key-chain or digest mismatches
Scope	FortiGate 7.4 / 7.6.
Solution	Background TCP Authentication Option (TCP-AO) enhances BGP session security by replacing TCP-MD5 with modern, stronger cryptographic methods. FortiOS supports TCP-AO starting with version 7.4.2 and uses key chains with HMAC-SHA1, HMAC-SHA256, or CMAC-AES128 algorithms. However, interoperability between vendors requires precise alignment of: Cryptographic algorithm. Key-chain configuration. TCP option inclusion. Key lifetimes. AO MAC behavior. Any mismatch causes the peer to silently drop SYN packets, preventing BGP from forming. FortiGate Configuration: Creating a Key-Chain for TCP-AO (FortiGate): Example using HMAC-SHA256: config router key-chain     edit "BGP-AO-CHAIN"         config key             edit "1"                 set accept-lifetime 00:00:00 01 01 2024 2147483646                 set send-lifetime 00:00:00 01 01 2024 2147483646                 set key-string SuperSecretKey123                 set algorithm hmac-sha256             next         end     next end Example using AES-128 CMAC: config router key-chain     edit "AES-CMAC-CHAIN"         config key             edit "1"                 set accept-lifetime 01:01:01 01 01 2021 2147483646                 set send-lifetime 01:01:01 01 01 2021 2147483646                 set key-string **********                 set algorithm cmac-aes128             next         end     next end Applying the Key-Chain to a BGP Neighbor: One command links the key-chain to the BGP Neighbor: config router bgp     set as 65000         config neighbor             edit "192.0.2.1"                 set auth-options "BGP-AO-CHAIN"             next         end     end FortiGate automatically assigns send/receive identifiers internally - there is no manual ID configuration on FortiOS. Verifying TCP-AO on a FortiGate: diagnose sys tcp-auth-options The command confirms: The selected algorithm (AES128 / SHA1 / SHA256). The active key. The lifetime values. The peer address. Cisco Configuration: Key-Chain for TCP-AO: key chain BGP-AO-CHAIN tcp key 1 cryptographic-algorithm hmac-sha-256 key-string 7 SuperSecretKey123 send-id 10 recv-id 10 accept-lifetime 00:00:00 Jan 1 2025 infinite send-lifetime 00:00:00 Jan 1 2025 infinite Bind the key-chain to a BGP Neighbor. router bgp 65000 address-family ipv4 vrf FW_TRAN neighbor 192.0.2.254 remote-as 65000 neighbor 192.0.2.254 ao BGP-AO-CHAIN neighbor 192.0.2.254 fall-over bfd single-hop neighbor 192.0.2.254 activate exit-address-family Ensure TCP Options are included in AO MAC Some Cisco platforms do not include MSS, SACK, timestamps, or window-scaling in the AO digest unless explicitly configured. This will cause errors such as: AO invalid current key. bad seg - AO invalid current key. Invalid current key – aborting key creation. Enforce option inclusion: tcp ao include-options This is required on multiple Cisco platforms for interoperability with FortiGate. Juniper Configuration: set routing-options tcp-session authentication algorithm hmac-sha-256 set routing-options tcp-session authentication key-chain BGP-AO-CHAIN key 1 key "SuperSecretKey123" set routing-options tcp-session authentication send-id 10 set routing-options tcp-session authentication recv-id 10 set protocols bgp group FGT type internal set protocols bgp group FGT local-address 192.0.2.1 set protocols bgp group FGT neighbor 192.0.2.254 peer-as 65000 set protocols bgp group FGT neighbor 192.0.2.254 tcp-session authentication key-chain BGP-AO-CHAIN Troubleshooting: why TCP-AO BGP sessions fail to establish. Peer silently drops SYN packets. Symptoms: FortiGate repeatedly sends SYN with AO. No SYN-ACK or RST received. BGP stays in Connect / Idle. Cause: AO MAC mismatch -> peer discards SYN. Algorithm compatibility problems: Common platform behavior: Some routers support only HMAC-SHA1. Some support HMAC-SHA1 + SHA-256. Some do not support SHA-256 and silently drop packets. Some platforms accept AES-128 CMAC only in specific software versions. Resolution: If in doubt, configure HMAC-SHA1 on all devices. TCP Options not included in AO Computation: If one peer includes TCP: MSS. Timestamps. SACK. Window scaling. If the other peer does not, AO digests differ. Symptoms: AO invalid key. Invalid MAC. Idle/Connect looping. No SYN-ACK. Fix: tcp ao include-options Key-Chain Not Activated / TCB Not Initialized: Symptoms: tcb is not available. Active open failed. Cause: Key-chain not bound to AO subsystem: Resolution: Recreate key-chain Reapply to BGP Verify TCB is active afterwards Key-Chain lifetime or ID mismatches: AO requires: Overlapping lifetimes. Matching IDs on non-FortiGate devices. Same algorithm. Same key-string. Any mismatch prevents TCP session establishment. Interoperability best practices: Verify algorithm support across all platforms. When uncertain, use HMAC-SHA1. Ensure overlapping key lifetimes. Match key strings exactly. Ensure IDs match on Cisco/Juniper. For Cisco, always evaluate: tcp ao include-options Verify SYN packets contain the same TCP options. Conclusion: TCP-AO significantly enhances BGP security, but successful operation in multi-vendor environments requires strict alignment of algorithms, key parameters, and TCP option handling.