| Description |
This article describes how to resolve packet loss issues in the SD-WAN overlay network. |
| Scope | FortiGate, SD-WAN. |
| Solution |
Consider a scenario where an administrator deploys an SD-WAN overlay with Remote internet access (RIA) for the branch network.
Packet loss in the tunnel interfaces of the SD-WAN member are most likely to occur repeatedly. View the status of the tunnel from the dashboard under SD-WAN -> Performance SLAs.
With the uplinks being affected by the packet loss, all underlying services such as voice over IP (VoIP), video streaming, and other critical banking services of the branch network are impacted in the process. It is possible to execute a ping command for packet loss verification through the CLI:
execute ping-option source x.x.x.x <- Tunnel IP. execute ping-option repeat-count 1000 execute ping-option adaptive-ping enable execute ping x.x.x.x PING x.x.x.x (x.x.x.x): 56 data bytes 64 bytes from x.x.x.x: icmp_seq=999 ttl=51 time=54.0 ms 64 bytes from x.x.x.x: icmp_seq=999 ttl=51 time=54.0 ms 64 bytes from x.x.x.x: icmp_seq=999 ttl=51 time=54.0 ms
--- x.x.x.x ping statistics --- 1000 packets transmitted, 1000 packets received, 4% packet loss round-trip min/avg/max = 53.3/53.9/73.4 ms 1. FEC (Forward Error Correction).
Resolve this issue by disabling the hardware offload feature and enabling forward error correction (FEC) on the IPsec inferences of both the Branch and HQ FortiGates. config vpn ipsec phase1-interface edit <Tunnel-Intf> set npu-offload disable set fec-egress enable set fec-ingress enable next end config firewall policy edit 0 set fec enable next end With FEC (forward error correction) enabled on the tunnel interface, the recipient receives a redundant packet to reconstruct any packet loss or any packets that arrived with errors.
 After enabling FEC on the interface, default FEC settings are applied to the tunnel. See the documentation for more information.
config vpn ipsec phase1-interface edit <Tunnel-Intf> set npu-ofload disable set fec-egress enable set fec-ingress enable set fec-redundant 1 (Default=1, Min=1, Max=5) set fec-base 10 (Default=10, Min=1, Max20) set fec-send-timeout 5 (Default=5, Min=1, Max=100) set fec-receive-timeout 50 (Default=50, Min=1, Max=1000) next end
Here, FEC works by sending a fec-redundant number of parity packets for every fec-base number of packets sent within the fec-send-timeout period.
However, if fec-send-timeout is reached, FortiGate sends the parity packets regardless of fec-base. The fec-send-timeout period starts after FortiGate receives the first packet destined for the tunnel.
The higher the value of fec-redundant, the more bandwidth is used.
While Forward Error Correction (FEC) does enhance the reliability of overlay networks and elevates bandwidth consumption within an IPsec tunnel, the hardware offload disabled traffic increases the CPU usage of FortiGate.
This FEC feature does not work with third-party vendor IPsec gateways.
To check the FEC status of the VPN Tunnel, run the following:
diagnose vpn tunnel fec <Tunnel_Intf>
From v7.0.2, it is possible to define the FEC profile for FEC to work with network SLA metrics. In this case, SD-WAN Tunnel overlay, Performance SLA, and SD-WAN Rule are maintained.
config system sdwan set status enable config zone edit "SD-WAN_Overlay" next end config members edit 1 set interface "TNL_1" set zone "SD-WAN_Overlay" next edit 2 set interface "TNL_2" set zone "SD-WAN_Overlay" next end end
In Performance SLA, ensure that the VPN overlay participants (TNL1 & TNL2) are included into the health-check process (Performance SLA participant).
This SLA health-check is used to monitor the quality of the VPN overlay. This performance SLA will be used in the tunnel interface later.
config health-check edit "Performance_SLA" set server "172.16.16.1" set members 1 2 config sla edit 1 next end next end
In the SD-WAN Rule, set an overlay interface as the priority member.
config service edit 1 set name "SD-WAN_Rule" set mode load-balance set dst "all" set src "all" config sla edit "Performance_SLA" set id 1 next end set priority-members 1 2 set priority-zone "SD-WAN_Overlay" next end
Now, configure the FEC profile for mapping FEC to bind with network SLA metrics, the FEC base, and redundant packets.
config vpn ipsec fec edit "FEC-Profile" config mappings edit 1 set base 5 set redundant 2 set packet-loss-threshold 4 next end next end
Mappings are considered a match when packet loss exceeds 4%. In such cases, FEC is implemented with five base packets and two redundant packets.
Now, attach the SD-WAN health check (Performance SLA) and FEC profile (FEC mapping) to the IPsec tunnel interface.
config vpn ipsec phase1-interface edit <Tunnel-Intf> set npu-offload disable set fec-egress enable set fec-ingress enable set fec-health-check "Performance_SLA" set fec-mapping-profile "FEC-Profile" set fec-redundant 1 (Default=1, Min=1, Max=5) set fec-base 10 (Default=10, Min=1, Max20) set fec-send-timeout 5 (Default=5, Min=1, Max=100) set fec-receive-timeout 50 (Default=50, Min=1, Max=1000) next end
The FEC profile parameters, such as FEC base and FEC redundant values, are implemented only if the SLA link quality surpasses a 4% packet loss threshold. Otherwise, the default FEC settings are utilized.
Check the FEC status of the VPN Tunnel. Example output is provided below.
diagnose vpn tunnel fec <Tunnel_Intf> egress: enabled=1 base=5 redundant=2 codec=0 timeout=10(ms) encode=0 encode_timeout=0 encode_fail=0 tx_data=0 tx_parity=0 ingress: enabled=1 timeout=50(ms) fasm_cnt=0 fasm_full=0 ipsec_fec_chk_fail=0 complete=6638 rx_data=8766 rx_parity=6638 recover=0 recover_timeout=0 recover_fail=0 rx=8766 rx_fail=0
Status of FEC when the threshold is not exceeded:
config vpn ipsec phase1-interface edit <Tunnel-Intf> set npu-offload disable set fec-egress enable set fec-ingress enable set fec-health-check "Performance_SLA" set fec-mapping-profile "FEC-Profile set fec-redundant 1 (Default=1, Min=1, Max=5) set fec-base 10 (Default=10, Min=1, Max20) set fec-send-timeout 5 (Default=5, Min=1, Max=100) set fec-receive-timeout 50 (Default=50, Min=1, Max=1000) next end
Check the FEC status of the VPN Tunnel. Example output is provided:
diagnose vpn tunnel fec <Tunnel_Intf> egress: enabled=1 base=10 redundant=1 codec=0 timeout=10(ms) encode=0 encode_timeout=0 encode_fail=0 tx_data=0 tx_parity=0 ingress: enabled=1 timeout=50(ms) fasm_cnt=0 fasm_full=0 ipsec_fec_chk_fail=0 complete=6824 rx_data=9173 rx_parity=6824 recover=0 recover_timeout=0 recover_fail=0 rx=9173 rx_fail=0
The default FEC base and redundant values are used when the link quality has not exceeded the limits specified in the FEC profile mapping.
Additionally, if fec-codec is set to xor, the base and redundant packet values will not be updated.
FEC is one of the effective methods for decreasing data loss in IPsec tunnels. However, it does not harness the capabilities of multiple IPsec tunnels and is not compatible with hardware offloading or connections that are not IPsec.
2. Packet Duplication.
Packet duplication is an SD-WAN feature that makes it possible to send duplicate packets through up to three additional members of any kind, provided the best route to the destination is an SD-WAN member and the links used for duplication have a route to the destination.
Unlike the parity packets in FEC, which are not exact copies of the original packets, the duplicate packets are verbatim copies of the original packets. This way, the duplicate packets can be used for data loss protection and for out-of-band inspection or packet capture.
Packet de-duplication can also be enabled on the receiving FortiGate. When it is enabled, the receiving FortiGate accepts only the first copy of the packet received and drops the additional copies. This serves to save resources at the receiving end by instructing FortiGate to forward only one copy instead of forwarding all copies and letting the next hop discard the additional packets.
When duplication rules are implemented, packets are replicated on other functioning links within the SD-WAN zone and then de-duplicated upon reaching the destination FortiGate.
There are two options:
For more information, see the administration guide. |
