Description
This article provides a complement to the following documentation and discusses known issues regarding asymmetric traffic flow in this configuration: Azure vWAN with FortiGate Network Virtual Appliances (aka Azure vWAN SD-WAN NGFW Deployment).
Scope
FortiGate-VM, Azure vWAN.
Solution
In an Azure vWAN environment with FortiGate Network Virtual Appliances (NVAs), the general topology is as pictured:
With this configuration, users located at a Branch site will be able to reach Azure-protected resources through the following flow:
- Users send traffic to the Branch FortiGate destined for an Azure resource.
- The Branch FortiGate sends traffic over one of the redundant IPsec tunnels established to the NVA FortiGate-VMs.
- The NVA FortiGate-VM receives the traffic and routes it towards the Azure Internal Load-Balancer (ILB).
- The Azure ILB forwards traffic toward the virtual host in the Azure VNet.
- The virtual host sends reply traffic back to the Azure ILB.
- The Azure ILB forwards traffic back towards one of the NVA FortiGate-VMs, which then forwards traffic back to the Branch FortiGate via IPsec tunnel (which ultimately reaches the originating client).
The final step is where issues related to asymmetric traffic can occur. Generally speaking, Azure ILBs do not guarantee symmetric traffic flow, which means that Azure can receive the original traffic from one NVA FortiGate-VM and send the reply traffic to a different NVA FortiGate-VM. For example:
- Original direction: Client -> Branch FortiGate -> NVA #1 -> Azure ILB -> VM Host.
- Reply direction: VM Host -> Azure ILB -> NVA #2 -> Branch -> Client.
In most scenarios, this does not cause a problem. As per the deployment guide, the NVA FortiGates will be configured with FortiGate Session Life Support Protocol (FGSP) and session synchronization, which together natively support asymmetric traffic flows (i.e. traffic will not be interrupted or dropped by the NVAs).
The Branch FortiGates can also tolerate asymmetric traffic flow in this scenario since they will have IPsec tunnel connections to each NVA FortiGate as well as equal-cost multi-path (ECMP) routes provided by BGP (i.e. reply traffic can be accepted on any of the IPsec tunnels going to the NVAs). Refer to the following documentation for more information on how the FortiGate handles asymmetric traffic flow when ECMP routes are utilized: (FortiOS Admin Guide) Controlling return path with auxiliary session.
However, some applications may be sensitive to asymmetric traffic flow (particularly if it impacts communication to servers hosted in the Azure environment), which can result in intermittent disruptions in the user experience. Additionally, asymmetric traffic flow in general can hurt network performance (e.g. preventing hardware acceleration on the FortiGate), and so it is generally undesirable.
With that in mind, there are two methods available to address asymmetry in this design:
Method 1: Enabling Source NAT on the NVA FortiGate Firewall Policies:
As noted above, the asymmetric traffic flow can occur because Azure's internal load-balancer can send reply traffic to a different NVA FortiGate-VM than the one that originated the traffic. With that in mind, it is possible to apply SNAT in the IPsec -> Azure LAN related Firewall Policies on the NVA FortiGates, which results in the Azure ILB seeing traffic sourced from the NVA FortiGate's own IP address, rather than the IP address of the end-client at the Branch site. This forces the Azure ILB to return traffic to the same NVA FortiGate that originated it, thus guaranteeing symmetric traffic flow.
Notably, this solution is low-cost and quick to implement, though the only downside is that traffic will now appear to be sourced from the NVA FortiGate's address (from the VM hosts' perspective). This might make traffic logging on the more confusing on the VM hosts' side (though the logs will be clear to understand on the FortiGates).
Method 2: Implement the UTM Redirection feature of FGSP on the NVA FortiGates:
In an FGSP cluster, it is possible to perform unified threat management (UTM, aka security) inspection on traffic passing through the FortiGate peers, but a key caveat is that traffic can be inspected by the original owner of the session (i.e. if traffic passes through NVA #1 then the reply traffic must also flow through NVA 1 for inspection to occur). Given that FGSP clusters are designed to support asymmetric traffic flow, a solution was created to allow FGSP peers to redirect traffic back towards the original owner of the session, which satisfies the aforementioned requirement.
It is possible to use the FGSP UTM Redirection feature so that traffic is redirected from one NVA FortiGate-VM to another, which helps to solve the asymmetry issue in this scenario (at least from the perspective of the Branch FortiGate). Consider the earlier example where the originating traffic arrives on NVA 1 but the reply traffic is received by NVA 2. In this scenario, NVA #2 can redirect the reply traffic to NVA 1, which inspects the traffic before forwarding it back towards the Branch FortiGate.
To enable this feature, simply add any security inspection profile to the Firewall Policies on the NVA FortiGates (e.g. add any combination of IPS, Application Control, Web Filtering, Antivirus, etc.). Either SSL Certificate Inspection or Deep Inspection may be used in these policies alongside the aforementioned security inspection profile. For more information, refer to the following sections from the FortiOS Administration Guide:
UTM inspection on asymmetric traffic in FGSP
UTM inspection on asymmetric traffic on L3
FGSP support for failover with asymmetric traffic and UTM
Note that in this particular scenario (i.e. using FGSP with FortiGate-VMs in Azure), additional care is required since the FortiGate-VMs will not have direct Layer 2 network connectivity between them. To work around this, UTM Redirection can be done via L3 FGSP connectivity (see above) or via VXLAN as suggested by the Azure vWAN SD-WAN NGFW Deployment Guide:
Appendix C - Dynamic rerouting
