Description
This article discusses a routing issue that can occur when implementing ADVPN with OSPF (this issue is irrelevant to ADVPN with BGP). More specifically, the issue can occur if there are at least three Spoke FortiGates that have established ADVPN shortcuts together in a chain (i.e. Spoke_A has a shortcut to Spoke_B, and Spoke_B has a shortcut to Spoke_C).
Scope
FortiGate, ADVPN with OSPF only.
Solution
As a quick reminder, ADVPN can use OSPF instead of BGP to handle dynamic routing advertisements between the network's various Hub(s) and Spokes. These two protocols take different approaches to the task of sharing routes between Hubs and Spokes:
- ADVPN with BGP uses iBGP Route-Reflection so that Spokes only ever form BGP peerings to the Hub FortiGate. The Hub then reflects these routes to the rest of the ADVPN Spoke network so that all Spokes know how to reach one another without forming a full-mesh peering network.
- When shortcuts are formed in this scenario, the BGP routes themselves do not change (e.g. network 10.0.0.0/8 reachable via next-hop 192.168.0.101) but the path to reach the next-hop does change (i.e. next-hop 192.168.0.101 is reachable via the shortcut tunnel connected route, rather than via a route to the Hub FortiGate).
- On the other hand, ADVPN with OSPF results in OSPF neighborships being formed whenever there are network links between FortiGates. At the start, Spokes will only have adjacencies with the Hub FortiGate, but once a shortcut tunnel is formed the Spoke will now also have adjacencies to the other Spoke(s).
- Once these Spoke-to-Spoke adjacencies are formed, it becomes possible for Spokes to directly advertise routes to one another, rather than receiving these routes exclusively through the Hub. Cost will be used to determine the best/shortest path to a given destination, and generally, it is a lower/better cost to go directly from Spoke to Spoke rather than go Spoke-Hub-Spoke.
Understanding the Problem.
With the above in mind, consider the following example topology:
- Hub FortiGate has a single ADVPN with OSPF tunnel configured. Spoke_A, Spoke_B, and Spoke_C each connect to that tunnel.
- In this starting state, each Spoke will have a single OSPF adjacency that corresponds to the Hub connection, and all paths to reach other Spokes will require traveling through the Hub (i.e. Spoke-to-Hub-to-Spoke flow).
- For this scenario, assume that the OSPF cost for the ADVPN tunnel interface is set to 1 for each FortiGate.
- Later, Spoke_A sends traffic to Spoke_B and forms a shortcut tunnel. Since they are directly connected, they form an OSPF adjacency and share routes between one another.
- This is beneficial, as Spoke_A will now have a direct route to reaching Spoke_B's advertised networks. These direct routes will have a lower OSPF cost compared to traveling through the Hub (Spoke-Spoke cost of 1 vs Spoke-Hub-Spoke cost of 2), and so Spoke_A will send traffic to Spoke_B via the shortcut tunnel. Things are working well so far.
- Spoke_C also sends traffic to Spoke_B later and forms a shortcut tunnel as well. Spoke_B shares its own learned routes to Spoke_C, and these include the new routes that it has learned from Spoke_A as well. At the same time, Spoke_B is also sharing the new Spoke_C routes to Spoke_A since they are adjacent neighbors.
- At this point, Spoke_A wants to send traffic to Spoke_C, and there are now two possible paths to take:
- Spoke_A has the option of sending traffic to Spoke_C via the Hub. This has an OSPF cost of 2.
- Alternatively, Spoke_A also has a route to Spoke_C via Spoke_B. This also has an OSPF cost of 2.
- Depending on ECMP routing, Spoke_A may decide to send traffic to Spoke_C via Spoke_B instead of the Hub. Unless Spoke_B has been configured with Firewall Policies to allow this IPsec-to-IPsec traffic (which is unlikely since this is an unplanned traffic flow), traffic will be dropped once Spoke_B receives it.
As shown above, the issue occurs because of the following key points:
- OSPF will allow for paths to form that bypass the Hub FortiGate, as adjacencies can be formed if OSPF is enabled and direct connectivity exists (OSPF is enabled on the parent IPsec tunnel interfaces and will be inherited by the shortcut child tunnels).
- OSPF costs for the Spoke-Hub-Spoke path can become identical to the Spoke-Spoke-Spoke path.
- If both paths exist, then there is a significant possibility that traffic will go in the wrong direction and be dropped.
- Even if Spoke_B did allow IPsec-to-IPsec traffic, it would defeat the purpose of having Spoke-to-Spoke shortcut tunnels and would impose unnecessary load on the Spoke_B FortiGate.
Resolving the Problem.
To resolve this issue, OSPF costs must be adjusted in a way such the following is true:
- Spoke-Spoke Cost must be less than the Spoke-Hub-Spoke Cost, otherwise, Shortcuts will not be utilized.
- Spoke-Hub-Spoke Cost must be less than the Spoke-Spoke-Spoke Cost, otherwise, shortcuts will not form and spokes will be subjected to unnecessary traffic (that will be dropped in all likelihood).
- To meet these needs, the following assumptions must be made:
- All OSPF Spokes will have the same Cost applied to their OSPF Interface entry for the IPsec tunnel (for consistency/ease of calculation).
- It is not possible to set different Costs for the Spoke-Hub IPsec tunnel vs. the Spoke-Spoke shortcut tunnel.
- Hubs will have a lower OSPF interface cost than Spokes (since Spoke-Hub-Spoke paths must be preferred over Spoke-Spoke-Spoke paths).
For instructions on adjusting OSPF Cost using OSPF Interfaces on the FortiGate, refer to the following Community KB Article: Technical Tip : Controlling OSPF routes in case of parallel (redundant) links - cost on OSPF interfa....
To calculate the appropriate OSPF cost for a single-tunnel ADVPN scenario**, use the following formula:
S1 < S1 + H1 < S1 + S1 (aka Direct Spoke-Spoke Shortcut < Spoke-Hub-Spoke < Spoke-Spoke-Spoke), where:
- S1 = The OSPF Cost assigned on the Spoke VPN tunnel interface.
- H1 = The OSPF Cost assigned on the Hub VPN tunnel interface.
This equation can be reduced further to the following: 0 < H1 < S1, indicating that the Hub's OSPF cost must simply be greater than 0 but less than the Spoke's OSPF cost.
** Note that this evaluation becomes substantially more-complex if there are multiple ADVPN tunnels in the configuration (for example, Hub FortiGate has two or more ADVPN tunnels that Spokes connect to and can form cross-overlay shortcuts with), and it also depends on whether or not the tunnels are tiered (for example, Primary/Secondary/Tertiary duty) or if they are equal to each other.
With that in mind, a simplified formula to use in these scenarios is as follows:
H(x) = 1 + (T*(x-1)), and,
S(x) = 1 + (T*x), where:
T = number of Hub Tunnels in topology.
H(x) = Cost for Hub Tunnel x (H1, H2, H3, etc.).
S(x) = Cost for Spoke Tunnel X (S1, S2, S3, etc.).
This formula starts by assigning that the starting OSPF cost assigned to the first Hub Tunnel (H1) is 1, then calculates evenly-spaced Costs based on the number of tunnels in the scenario.
These costs work out such that there is a clear order of preference for which path should be taken when there are multiple options, though it relies on the principle that tunnels are only used in a clear order of preference (i.e. use Tunnel 1 primarily, use Tunnel 2 as Secondary Backup only, and use Tunnel 3 as Tertiary Backup only. For example, a three-tunnel setup would have the following resultant OSPF costs:
- Hub-side Tunnels 1, 2, and 3 - Cost of 1, 4, and 7.
- Spoke-side Tunnels 1, 2, and 3 - Cost of 4, 7, 10.
Given the complexity of using ADVPN with OSPF when multiple tunnels come into play, it may be a good idea to consider switching to ADVPN with BGP instead. BGP requires admins to specify the neighborships that can be established (which helps to eliminate the parallel-path complexity inherent to OSPF) and it also has access to a substantially greater number of path filtering/control tools that make it easier to tune for specific environments.
