Description
This article describes how to understand how FortiGate achieves non-stop forwarding under HA and Graceful restart capability.
Scope
FortiGate running in NAT and HA mode.
Solution
Diagram:
FortiGate HA-------BGP Peer----Router(Graceful restart capable).
To achieve non-stop forwarding and avoid packet drops in the network, FortiGate employs HA and Graceful restart capability.
BGP peers advertise the GR capability in the OPEN message when the BGP session is being set up.
The idea is that when a failover happens on the FortiGate side, tell the BGP peer router that there is a FortiGate restart event.
Because of this, the GR-capable peer router is required to keep the FIB information and continue forwarding traffic for the configured graceful-restart-timer. The new Primary can use this time to set up a new BGP session and switch to using the newly learned routes.
During normal operation, the secondary unit in HA receives FIB information from the primary. This information is visible in the kernel table as routes with high-priority.
FGT2 # get router info kernel | grep 200.200.200.200
tab=254 vf=0 scope=0 type=1 proto=18 prio=2147483649 0.0.0.0/0.0.0.0/0->200.200.200.200/32 pref=0.0.0.0 gwy=10.80.3.112 dev=4(port2) <----- HA synced routes.
When a failover happens, the new primary FortiGate initiates a new TCP connection with the BGP peer and sets the restart flag as 1 in the graceful restart capability flag.
2023-05-22 13:06:21 BGP: [NETWORK] Accept Thread: Incoming conn from host 10.80.3.111 (FD=28 VRF=0)
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [FSM] State: Established Event: 14
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [GRST] Initial Announce: Stalepath Preserve Timer(360) started
2023-05-22 13:06:21 id=20300 msg="BGP: %BGP-5-ADJCHANGE: VRF 0 neighbor 10.80.3.111 Down Unexpected TCP state change"
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [GRST] Action Established: Restart Timer(120) started, Event 14
2023-05-22 13:06:21 BGP: [GRST] Timer Announce Defer: Check
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [FSM] State: Connect Event: 14
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [FSM] InConnReq: Accepting...
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [NETWORK] FD=28, Sock Status: 0-Success
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [FSM] State: Connect Event: 17
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [ENCODE] Msg-Hdr: Type 1
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [ENCODE] Open: Ver 4 MyAS 200 Holdtime 180
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [ENCODE] Open: Msg-Size 71
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [DECODE] Msg-Hdr: type 1, length 71
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [DECODE] Open: Optional param len 42
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [DECODE] Open Opt: Option Type 2, Option Len 6
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [DECODE] Open Cap: Cap Code 1, Cap Len 4
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [DECODE] Open Opt: Option Type 2, Option Len 6
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [DECODE] Open Cap: Cap Code 1, Cap Len 4
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [DECODE] Open Opt: Option Type 2, Option Len 2
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [DECODE] Open Cap: Cap Code 128, Cap Len 0
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [DECODE] Open Cap: RR Cap(old) for all address-families
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [DECODE] Open Opt: Option Type 2, Option Len 2
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [DECODE] Open Cap: Cap Code 2, Cap Len 0
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [DECODE] Open Cap: RR Cap(new) for all address-families
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [DECODE] Open Opt: Option Type 2, Option Len 6
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [DECODE] Open Cap: Cap Code 65, Cap Len 4
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [DECODE] Open Opt: Option Type 2, Option Len 8
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [DECODE] Open Cap: Cap Code 64, Cap Len 6
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [DECODE] Cap GR: Restart Flag On, Restart Time 120
2023-05-22 13:06:21 BGP: 10.80.3.111-Outgoing [DECODE] Cap GR: AFI/SAFI 1/1 Fwd-state Flag 1, action: Set
On seeing this flag, the peer router believes that his BGP peer i.e. the FortiGate HA is undergoing a restart.
Since the router is GR capable, it will hold the FIB information and continue forwarding traffic until BGP converges again (provided BGP converges before the stale path timer expires).
- The new primary will continue forwarding packets using the FIB that it had learned through HA until the duration of route-ttl.
- After BGP has converged, new learned routes will be installed in the FIB with lower priority (usually 1).
- There is a possibility of seeing both the HA synced routes and the new BGP learned routes together in the FIB when a higher route-ttl value is configured.
FGT2 # get router info kernel | grep 200.200.200.200
tab=254 vf=0 scope=0 type=1 proto=11 prio=1 0.0.0.0/0.0.0.0/0->200.200.200.200/32 pref=0.0.0.0 gwy=10.80.3.112 dev=4(port2) <----- Route learned through BGP on new primary.
tab=254 vf=0 scope=0 type=1 proto=20 prio=2164260865 0.0.0.0/0.0.0.0/0->200.200.200.200/32 pref=0.0.0.0 gwy=10.80.3.112 dev=4(port2) <----- HA learned routes that were being used to forward traffic.
- HA routes will disappear after the configured duration of route-ttl.
FGT2 # get router info kernel | grep 200.200.200.200
tab=254 vf=0 scope=0 type=1 proto=11 prio=1 0.0.0.0/0.0.0.0/0->200.200.200.200/32 pref=0.0.0.0 gwy=10.80.3.112 dev=4(port2)
For more details about the configuration around GR BGP, refer to the following KB article:
Technical Tip: Configuring FortiGate HA and BGP graceful-restart to avoid traffic interruption durin...
