Technical Tip: Configuring FortiGate HA and BGP graceful-restart to avoid traffic interruption during an HA failover
Description
This article describes the configuration that needs to be applied to a FortiGate HA cluster and the BGP settings so that each router (the FortiGate and its peer(s)) will keep the BGP routes in their routing table(s) to avoid traffic interruption during an HA failover.
Scope
FortiGate running in NAT and HA mode.
Solution
Expectations, Requirements:
The HA cluster with one or more BGP peers will failover without traffic interruption.
Configuration:
On a FortiGate HA cluster, the BGP router daemon process is only running on the Primary unit. When there is an HA failover, a new BGP process will be launched on the newly elected primary.
Even though the FortiGate has all the routes, if the peer sees the FortiGate as unresponsive, it will remove all the routes from its routing table, and traffic will be interrupted:
Therefore, what needs to be done to avoid traffic interruption can be divided into three parts, which are detailed later:
Check that the remote peer will not delete the routes.
Check that the FortiGate cluster will keep the BGP routes in the routing table.
Fine-tune timers.
Check that the remote peer will not delete the routes. This can be achieved with BGP graceful restart. 'Graceful Restart', known as Cap. Code 64, is a BGP capability. It is an Internet standard defined in RFC 4724 and it is used by a BGP peer to indicate its ability to preserve its Routing Table (Forwarding State) during a BGP restart. It can also be used to convey to its peer its intention of generation of the End-of-RIB marker upon the completion of its initial routing updates. This capability needs to be configured on both peers.
Note:
The End-of-RIB market, known as EoR, is a BGP UPDATE message with no reachable Network Layer Reachability Information (NLRI) and it can be used by the BGP speaker to indicate to its peer the completion of the initial routing update after the session is established. This means that the last BGP UPDATE message we received includes the EoR Marker. In a packet capture, this is printed in a BGP UPDATE packet with a multiple "f" Marker and a length of 23.
On FortiGate, Graceful Restart can be enabled at both the:
Global configuration level.
Peer level.
Note:
Enabling Graceful Restart is service-impacting. Enabling it at the global configuration level will cause all BGP peers to flap, and enabling it at the peer level will cause the single BGP peer to flap.
Configuration snapshot:
config router bgp
set as 65111
set graceful-restart enable
config neighbor
edit "10.2.3.4"
set capability-graceful-restart enable
set bfd enable
set remote-as 65000
set weight 20
next
end
config network
edit 1
set prefix 172.31.0.0 255.255.0.0
next
end
end
'set bfd enable' <-- BFD options will help to trigger it with milliseconds for OSPF or BGP connections: Use BFD with caution.
Graceful Restart Operation:
Based on the RFC 4724, when the Restarting Speaker restarts, it MUST retain, if possible, the
forwarding state for the BGP routes in the Loc-RIB and MUST mark them as stale. It must not differentiate between stale and other information during forwarding.
Once the session between the Restarting Speaker and the Receiving Speaker is re-established, the Restarting Speaker will receive and process BGP messages from its peers.
However, it MUST defer route selection for an address family until it either (a) receives the End-ofRIB marker from all its peers (excluding the ones with the "Restart State" bit set in the received capability and excluding the ones that do not advertise the graceful restart capability) or (b) the Selection_Deferral_Timer referred to below has expired. It is noted that prior to route selection, the speaker has no routes to advertise to its peers and no routes to update the forwarding state.
The “Selection_Deferral_Timer” in FortiGate case is the parameter called “graceful-update-delay” that we discuss later.
Important Note:
It is highly important to mention that BFD and Graceful Restart are two features that should not be both activated at the same time. This has not been recommended by all the Vendors. By activating both features on BGP, the routing performance will be sub-optimal. The reason behind that is related to the fact that BFD uses Data-Plane, whereas Graceful-Restart uses the Control-Plane. This operational difference makes implementing BFD alongside graceful restart an option that should be carefully considered, and, for that reason, it is not recommended.
Check that the FortiGate cluster will keep the BGP route in the FIB table. When the FortiGate is configured in an HA cluster, all the routes will be synchronized to the secondary devices. The synchronized routes on the secondary will have a limited lifetime and a lower priority. The lifetime of these routes can be configured through the 'route-ttl' parameter in the system HA configuration:
route-ttl:
Controls how long HA routes are kept in the FIB of a cluster unit after it has been promoted as Primary.
Increasing 'route-ttl' can help provide a seamless failover.
The route-ttl range is 5 to 3600 seconds. The default route-ttl time is 10 seconds.
route-wait:
The time the primary unit waits after receiving the routing table update before sending the update to the subordinate units in the cluster.
The route-wait range is 0 to 3600 seconds. The default route-wait is 0 seconds.
route-hold:
The time that the primary unit waits between sending routing table updates to subordinate units in a cluster.
The route hold range is 0 to 3600 seconds. The default route hold time is 10 seconds.
config system ha
set route-ttl 30
end
After the route times out, the new master will need to learn the route again through route convergence. The default value is 10. A route-ttl value of 30 or 60 seconds is recommended when we have Graceful Restart enabled and BGP convergence.
Fine-tuning timers. The following are the main timers that can be tuned:
holdtime-timer (default 180): Number of seconds to mark the peer as dead.
This is the number of seconds to wait between keepalive, update, or notification messages before considering the connection to the peer as closed.graceful-restart-time(default 120): Time needed for neighbors to restart (sec).
This is the number of seconds to wait for the OPEN message before removing the stale routes
graceful-restart-time should be less than or equal to the holdtime-timer.graceful-update-delay (default 120): Route advertisement/selection delay after restart. After an HA failover, the route populated on the new primary would be delayed based on the timer applied against this setting.
graceful-stalepath-time(default 360): Time to hold stale paths of restarting neighbor (sec). The total maximum time that a stale route should be kept before being deleted.
CLI Syntax:
config router bgp
set graceful-restart enable
set graceful-restart-time <integer value> --> graceful-restart-time, Enter an integer value from <1> to <3600> (default = <120>).
set graceful-stalepath-time <integer value> --> graceful-stalepath-time, Enter an integer value from <1> to <3600> (default = <360>).
set holdtime-timer <integer value> --> holdtime-timer, Enter an integer value from <3> to <65535> or (special = <0>) (default = <180>).
set graceful-update-delay <integer value> --> graceful-update-delay, Enter an integer value from <1> to <3600> (default = <120>).
end
Consider tuning these counters with the following two criteria:
Time to detect a real BGP peer failure.
Maximum time allowed for a restart.
Note:
When graceful-restart is enabled, it will delay the time at which a real network/peer failure will be detected, and as a consequence, this will end up in a downtime that can be as long as the graceful-restart-time.
Therefore, those timers must be configured to a value that suits the network requirements. Also, do not expect that after the failover is finished, BGP peering will continue to work with the same uptime as on the previous primary device. BGP will be re-established, so the BGP 'flap' will be visible. But this is expected behavior.
4. Another important parameter is the "set advertisement-interval". According to the BGP standard (RFC 4271), a BGP router should not send updated reachability information for an NLRI to a BGP peer until a specific period of time, Minimum Route Advertisement Interval (MRAI), has elapsed from the last update. By changing the advertisement interval to a value of 1 to 3 seconds, we are automatically optimized the convergence time of the BGP.
config router bgp
config neighbor
edit <neighbor ip>
set advertisement-interval 1
next
end
end
Verification.
In the output of the CLI commands:
FGT # get router info bgp neighbor a.b.c.dCheck the graceful restart capabilities :
For address family: IPv4 Unicast:
BGP table version 1, neighbor version 0
Index 1, Offset 0, Mask 0x2
AF-dependant capabilities:
Graceful restart: advertised, received
Check timers in the CLI command:
FGT # get router info bgp neighbor a.b.c.d Routes in the FIB can be validated by using the following command:
diagnose ip route listOr:
get router info kernel
Troubleshooting:
A packet capture taken with the BGP peer IP will be helpful.
Check other BGP-related information with:
FGT # get router info bgp neighbor
FGT # get router info bgp summary
FGT # get router info bgp network
FGT # get router info routing-table databaseIf a failover and subsequent failback occur within the time defined by the route-ttl parameter (configured via set route-ttl <seconds>), the event is classified as a route flap.
When testing failover, use 'diagnose sys ha reset-uptime' on the primary or adjust HA priority to simulate the normal HA Primary election process. See this article: Technical Tip: Different options to trigger an HA failover (FGCP). Do not use 'exec ha failover set'.
In the case of a route flap:
The affected routes are flushed from the routing table.
Associated sessions are removed after the system processes five packets following the route change.
