Troubleshooting Tip: How to solve SMB traffic slowdown over Fortinet SD-WAN

[stroia]

Description	This article describes how to troubleshoot slow SMB traffic transfer over FortiGate SD-WAN.
Scope	FortiGate SD-WAN.
Solution	SMB (The Server Message Block) is a client-server communication protocol using ports 139 and 445 with TCP. This protocol is used to provide access to files, printers, serial ports and other resources on a network, while ‍Samba is an open-source implementation of the SMB protocol. Typically, SMB is used to access a file hosted in a shared folder on a remote server.  The article provides a guide about how to troubleshoot when expiring slowness with download or upload files. A bottom-up troubleshooting approach following ISO Model is recommended.  Connections status analysis: on Fortinet SD-WAN, Underlays or Overlays connections must be mapped as SD-WAN Members.To discover a performance degradation on a SD-WAN member monitored with a Performance SLA, there are 3 ways: Via the FortiGate GUI: navigate to Network -> SD-WAN -> Performance SLA (third tab). Below is an example of a temporary 100% packet loss on 1 of 2 members monitored: Via the FortiGate CLI: run the following command: diagnose sys sdwan health-check Via FortiAnalyzer: Log View -> Fortigate -> Event -> SD-WAN: More detailed information can be found in this document: Administration guide: Monitoring performance-sla. In cases where IPsec tunnels are used as SD-WAN members, an underlay degradation can usually cause IPsec flaps. It is possible to verify IPsec tunnel stability by analyzing VPN Logs under Log & Report -> System Event -> VPN Events: Poor underlay performance is only one of the possible causes of IPsec instability. To find out the root cause, it is necessary to debug the ike demon responsible for IPsec tunnel installation and maintenance. More details can be found in these articles: Troubleshooting-Tip IPSEC Tunnel debugging IKE  Administration Guide: Using the debug flow tool  Troubleshooting-Tip: First steps to troubleshoot connectivity  SD-WAN Rules configuration analysis: to understand if the SD-WAN is properly configured, it is necessary to first verify which one is steering the traffic with the slowdown issues. Here are two ways to find the matching rule: Analyze traffic logs under Log & Report -> Forward Traffic (add the 'SD-WAN Rule Name' column by right-clicking on the column’s title row and selecting it) by applying the necessary filters: Use the following command with the proper filters: diagnose sys session list More details about how to do this are available here: Technical Tip: How to find the SD-WAN rule and SD-WAN member used. When analyzing SD-WAN logs, remember the following: For local-out and return traffic, SD-WAN rules are not matched by default, as explained in Technical-Tip: Use SD-WAN for local out traffic or Management and in Technical-Tip: Return Reply traffic is not matching policy routes. The traffic is steered from an SD-WAN rule if the validation rule mechanism is valid, as explained in Technical-Tip: Explaining the SD-WAN rule matching process. if traffic does not match any explicit SD-WAN rules, the implicit rule makes it forward according to the routing table (which will differ from the firewall policy matching order). One possible cause for the issue is if the rule that steers SMB traffic uses a Load Balance Strategy (called Maximize Bandwidth (SLA) prior to FortiOS v7.4.1 firmware), as in this example: This could be the cause of the issue. A huge amount of SMB traffic is managed by different sessions by FortiGate, so if the SD-WAN rule matched a used Load Balance strategy, traffic is equally distributed per session with a round-robin algorithm between all rule SD-WAN members in SLA (as explained in Administration Guide: Load balancing strategy) and shown on logs for a transfer file using SMB: It is possible to force traffic to use only the best member with a new rule moved on top of an existing one and configured with proper IP/subnets and a Best Quality strategy. For example: The SD-WAN Best Quality strategy chooses the best link to forward traffic comparing the 'link-cost-factor' defined via health-check, as shown in the following logs for a file transfer via SMB: More details about Best Quality strategy can be found here: Administration Guide: Best Quality strategy. The SD-WAN rule member used is the one with the black tick: If the issue is still not resolved after all of the analysis and tests described above, investigate the following: Firewall policy configuration (for example, it may be necessary to allow pinging for an ICMP client server to permit the client server probe mechanism to work). Packet fragmentation and IPsec tunnel drops, as explained here: Technical Tip: IP Packet fragmentation over IPsec tunnel To identify the root cause, it may be necessary to analyze traffic captures: Via the GUI (from FortiOS 7.2.0): Embed real time packet capture and analysis tool on diagnostics page Via the CLI: Troubleshooting-Tip Using the FortiOS built in packet sniffer Note the following: Offloaded traffic is not shown from the built-in traffic capture FortiOS tool on a FortiGate hardware model. It is necessary to disable offloading on the firewall policy (as explained in the Disabling np offloading for firewall policies documentation) and on the IPsec tunnel phase1 (as explained in the Disabling NP offloading for individual IPsec VPN phase 1 documentation) to see all traffic. When IPsec phase1 offload is disabled, all IPsec tunnels are flushed and a rekey process starts. While in progress, this operation affects traffic. When troubleshooting SMB bandwidth and throughput issues, latency and TCP Window Size are crucial factors in calculating the expected throughput.  The SMB protocol is designed for local file sharing with low latency and it transmits data in 'blocks' or packets, breaking it down into smaller, manageable chunks for transmission instead of sending a single large piece or stream. As a result, its efficiency may be questioned. Even a slight packet loss or delay in these exchanges can cause noticeable slowdowns. Therefore, it is highly sensitive to factors like packet loss, out-of-order packets, and latency. Even minor network issues can significantly impact its performance, particularly during file transfers and other operations that depend on quick responses. Other related documents: SD-WAN architecture for enterprise List of SD-WAN related diagnose commands IPsec tunnel drops and packet fragmentation: Technical-Tip: Troubleshooting IPsec VPN tunnel errors  Technical Tip: Behavior of TCP MSS setting under system interface  How hardware acceleration works Technical Tip: Low throughput troubleshooting