Technical Tip: Understanding the link-down-failover, fast-external-failover, and interface options for BGP

[pjang]

Description

This article describes the behavior and the usage of the link-down-failover, fast-external-failover, and interface options that exist for BGP on the FortiGate. These options affect BGP peering behavior in different but highly complementary ways, and thus this article will discuss each of them individually, followed by how they can be used together.

Scope

FortiGate, BGP.

Solution

1. Option location in CLI
2. link-down-failover
3. fast-external-failover
4. interface
5. Combining these options
6. Notes regarding BGP Convergence Times

Option location in CLI (* = default setting):

config router bgp

    set fast-external-failover [ enable* | disable ]

        config [ neighbor | neighbor-group ]

            edit <name>

                set link-down-failover [ disable* | enable ]

                set interface <interface_name, empty by default>

link-down-failover:

• This BGP option is configured on a per-neighbor/neighbor-group basis (under config neighbor / config neighbor-group) and is disabled by default.

• When link-down-failover is enabled, the FortiGate will dynamically monitor the outgoing interface used for each BGP neighborship. If the FortiGate detects that the outgoing interface has been brought down for some reason (e.g. physical link disconnection, administrative shutdown, VPN dead-peer detection, etc.,) then it will immediately bring down the BGP neighborships associated with that outgoing interface. The BGP routes corresponding to that remote peer will also be immediately removed from the routing table.

• When this option is not enabled, then the FortiGate will keep the BGP neighborships active until the BGP hold timer is exceeded (default 180 seconds or 3x keepalive timer).

• Example: FortiGate_A and FortiGate_B are connected by a site-to-site IPsec tunnel and have formed a BGP neighborship over the VPN. At some point, the two FortiGates lose connectivity to each other, causing IKE Dead-Peer Detection (DPD) to detect the issue and bring the tunnel down.
  • With link-down-failover enabled, each FortiGate can immediately bring down this BGP neighborship once the VPN tunnel goes down, and they can also remove the BGP routes associated with that neighbor from the routing table.
    • This can be highly-advantageous, as it can help to prevent traffic from being routed out an unexpected interface (see further below for further information).

• With link-down-failover disabled, each FortiGate will retain the existing BGP neighborships until the BGP hold timer is expired. The associated BGP routes will also be retained in the routing table.

This can be disadvantageous, as it can allow private traffic to be routed out the wrong interface (such as the WAN/Internet) due to the nature of BGP recursive route lookups.

• Consider a BGP route like '172.16.100.0/24 [20/0] via 192.168.1.2' on FortiGate_A (192.168.1.2 being the IP address of FortiGate_B's VPN tunnel interface.
  • When the VPN is online, FortiGate_A would have a connected (or static) route for 192.168.1.2 via the VPN interface, so BGP route-recursion would determine that 172.16.100.0/24 is ultimately reachable via the VPN tunnel.
  • If that VPN tunnel goes down, then it is very likely that the next-best match would be the default route going to the Internet (e.g. '172.16.100.0/24 [20/0] via 192.168.1.2 (recursive via 100.0.0.1, wan1)')

fast-external-failover:

• This option is configured in the main section for BGP configuration (config router bgp) and is enabled by default.

• When this option is enabled, the FortiGate will dynamically monitor the outgoing interface used for each BGP neighborship, similar to link-down-failover.
  • However, the main difference is that fast-external-failover is not used to handle BGP neighborships on the local FortiGate. Instead, when the FortiGate detects that the outgoing interface has been brought down (or if its IP addressing has been modified/deleted), it will send a BGP Notification message to the remote peer to instruct them to bring down their BGP neighborship.
  • Note that this feature applies to EBGP neighbors specifically, not IBGP.
  • This BGP Notification will have a CEASE error code with an UNSPECIFIED sub-error code, as per RFC 4271 Section 6.7 (https://datatracker.ietf.org/doc/html/rfc4271#section-6.7)
  • Note that this is a 'best-effort' feature, in that the BGP Notification sent by the FortiGate may not reach the remote peer (i.e. if the interface goes down for an extended period of time, or if network connectivity is entirely lost between the two peers).

• Example: FortiGate_A and FortiGate_B are connected by a site-to-site IPsec tunnel and have formed an EBGP neighborship over the VPN. During a maintenance window, the administrator restarts the VPN tunnel with diag vpn ike gateway clear on FortiGate_A.
  • With fast-external-failover enable, FortiGate_A will send a BGP Notification with error code CEASE once the VPN tunnel comes back online (because it has detected a change in the state of the VPN tunnel interface). This will instructs the remote peer (FortiGate_B) to close/restart the BGP peering to FortiGate_A.
  • With fast-external-failover disable, FortiGate_A will not send the BGP Notification, and remote FortiGate_B will not clear its existing BGP peering or timers.

• An alternative example could involve a FortiGate and an ISP router that peer via EBGP. If the FortiGate's outgoing interface has a change in IP address (perhaps due to DHCP or the administrator making the change), then the FortiGate can inform the ISP router to immediately clear the old neighborship and stop using potentially-stale routes.

interface:

• This option is configured on a per-neighbor/neighbor-group basis (under config neighbor / config neighbor-group) and is unset/disabled by default.

• This option allows an administrator to specify an outgoing interface for that BGP peer connection (as opposed to the FortiGate determining the outgoing interface via the routing table).
  • When the designated interface goes down, the FortiGate will not attempt to bring up the associated BGP neighborships using any other interface. The BGP peering will transition to the Idle state and stay there until the designated interface comes back online.
  • Notably, FortiGate will not bring down an existing BGP neighborship when the designated interface goes offline (i.e. interface does not perform the same duty as link-down-failover). The neighborship will remain Established (as will the associated BGP routes) until the Hold timer expires.

• Example: FortiGate_A and FortiGate_B are connected by a site-to-site IPsec tunnel and have formed a BGP neighborship over the VPN. During a maintenance window, the administrator administratively shuts down the VPN tunnel interface on FortiGate_A.
  • With interface enabled and set to the VPN tunnel interface, FortiGate_A will not send out any further BGP traffic towards FortiGate_B while the VPN tunnel interface remains down (e.g. it will not try to reach FortiGate_B's neighbor IP using a different interface).
  • If link-down-failover is not enabled, then FortiGate_A's BGP neighborship will remain Established until the Hold timer expires, after which the neighborship moves to the Idle state and will remain there until the VPN interface comes back up.
  • With interface disabled/unset, FortiGate_A will continue to send out BGP traffic (e.g. keepalives) using the next-best interface based on the route table (this is typically a WAN interface using the default Internet route). The BGP neighborship will remain in the Established state until the Hold timer expires, then it will transition to Idle and then Active as it attempts to re-establish the neighborship via a different interface.
  • The main benefit of using the interface option is that it prevents the FortiGate from sending BGP traffic for a specific neighbor out of the wrong interface. If the FortiGate reroutes BGP traffic to an incorrect outgoing interface for some reason (e.g. the VPN tunnel flaps briefly), then BGP re-establishment can become impacted until the Hold timer expires (i.e. an outage lasting longer than absolutely necessary).

Combining these options:

• The above BGP options each have unique functions, but they can be combined to make the FortiGate much more predictable during an outage scenario.

• Consider the previous examples, with two FortiGates (FortiGate_A and FortiGate_B) connected via an IPsec VPN tunnel. When all three BGP options are enabled, the following benefits can be realized:
  • With link-down-failover enabled, the FortiGate can detect when the VPN/interface goes offline and pull down the associated BGP neighborship/routes.
    • If these FortiGates have multiple BGP connections to each other (e.g. across multiple VPN tunnels, MPLS links, etc.) then removing the routes for the downed interface will allow them to quickly reroute to a better alternative.
    • If there is only the singular connection between the FortiGates, then removing the neighborship/routes prevents traffic from being accidentally routed out the wrong interface (e.g. the default WAN route).

• With fast-external-failover enabled, the FortiGate can inform its eBGP peer to restart the BGP connection based on state changes for the outgoing interface.

If the IP addressing of a VPN tunnel interface is changed (perhaps due to mode-config or DHCP addressing), then the FortiGate can inform the remote peer to restart the BGP connection and their timers. This can prevent traffic loss issues caused by stale/incorrect routes.

• With the interface option configured, the FortiGate can ensure that BGP traffic for a given neighbor will only ever be sent out using one specific interface.

This prevents the FortiGate from using non-viable interfaces to try and reach a BGP neighbor, and it can also help eliminate delays when re-establishing BGP neighborships after an outage occurs (since it ensures that the FortiGate does not get stuck trying to send BGP traffic out the wrong interface).

Notes regarding BGP Convergence Times.

• Traditionally, BGP convergence times are dictated by BGP's timers, such as the Keep-Alive, Hold-Down, and Advertisement Interval timers. These timers affect how quickly BGP detects neighborship failures, as well as how long BGP should wait between advertising/withdrawing routes to a given BGP peer.

• Likewise, BFD (Bidirectional Forwarding Detection) is another complementary protocol that can aid in improving BGP convergence times, as it can shutdown neighborships very quickly upon loss of network connectivity.

• However, care has to be taken when considering timer adjustments and enabling BFD, as too-aggressive settings can be more harmful than helpful to BGP stability.

• With that in mind, the options described in this article should be considered another set of tools that help reduce BGP convergence times, especially if modifying timers and enabling BFD are not appropriate for a given environment.
  • By enabling the BGP options described in this article, FortiGate can inform BGP more quickly when network failures occur. From there, BGP can bring down dead neighborships and withdraw routes, minimizing the amount of traffic that is sent down a dead network link.
  • This is especially true for BGP neighborships established over VPN tunnels, as the VPN interface itself is brought down when the tunnel loses network connectivity with its peer.