Troubleshooting Tip: How to investigate SIP calls dropping and SIP call quality issues

[saleha]

Description	This article provides investigation methods for call quality and random call drops issues on the FortiGate side. The focus here is on SIP calls as the most popular VOIP protocol.
Scope	FortiGate - FortiOS
Solution	Based on the nature of intermittent or event frequent call drops, audio quality, sudden loss of audio or similar problems, it logically indicates VOIP proxy is setup correctly as the calls do get established and running until one of these issue does occur. This however does not mean that VOIP proxy status and configuration to be ignored as they can provide indication of where the problem lies. The following SIP commands can provide information about SIP calls status and stats. Note that there are two modes of operation for SIP ALG in FortiOS: Kernel-helper-based mode: FortiGate acts as a a session helper offering basic SIP and RTP NAT as well as opening pinholes. Diagnostics: diagnose sys sip status diagnose sys sip dialog {clear | list} diagnose sys sip mapping list Proxy-based mode: This is the default SIP alg mode. Offers all the VOIP profile security features such as strict-register, pinholes, application layer NAT, SDP packets manipulation and more. Diagnostics: diagnose sys sip-proxy calls {clear | list | idle | invite} diagnose sys sip-proxy stats {clear | list} diagnose sys sip-proxy filter <options> diagnose sys sip-proxy log-filter <options> For reference on SIP ALG different features, the following link provides further details: SIP ALG and SIP session helper. Next step is to inspect this traffic running packet capture on both the incoming and outgoing interfaces on the Firewall simultaneously. The purpose is to investigate the actual SIP and RTP traffic traversing through the firewall specially that network analyzers such as Wireshark do offer the options to show traffic flow and playback captured calls.  To run a packet capture on FortiOS through the GUI, navigate to Network -> Diagnostics -> Create new capture. To run the capture from the CLI: diagnose sniffer packet <interface name> "host x.x.x.x and port 5060" 6 0 l <- Example of sniffer on an IP address and port 5060. For instructions on how to run a sniffer, see Troubleshooting Tip: Using the FortiOS built-in packet sniffer for capturing packets. In the event of an intermittent issue, firewall performance has to be taken into consideration. It is therefore recommended to check System Event logs under the Log & Report page, as well as Memory and CPU usage from the Dashboard -> Status page over the interval matching the incident time, and the Crash-log via the following CLI command: diagnose debug crashlog read One of the most common scenarios for intermittent SIP call issues is cases where SIP call traffic changes routes during the call due to an Equal Cost Load Balancing (ECMP)effect when multiple routes have the same distance and priority to destination. This issue can be easily identified using the sniffer command or from Forward Traffic logs where the destination interface for different sessions to the same SIP server destination is switching. The solution for the ECMP problem is prioritizing SIP traffic by changing priority or distance on the configured Static routes for example. Another solution is using policy-routes rules to dictate one destination interface only for this traffic. Policy route reference: Technical Tip: Configuring the Firewall Policy Routes. Another similar scenario is the use of SD-WAN configuration. SD-WAN is a solution usually used to steer egress traffic using criteria such as the health status of the egress interfaces by forcing those interfaces to send probe packets to a destination and measure their health based on the response. Alternatively, give preference to an interface manually as the primary choice as long as that SD-WAN member interface is deemed up. See this document for detailed information regarding an SD-WAN solution: SD-WAN components and design principles. SIP calls are sensitive to route change during RTP stream. If the SD-WAN member passing SIP call traffic breaches the health-check configured threshold, an SD-WAN rule configured to steer this traffic using that interface will force traffic to route out a different member causing the effect of audio loss or call drop. The solution should primarily focus on identifying and resolving why the health-check threshold was breached. As an example: a fix in the case of unexpected behavior from an intermittent ISP service may be as simple as tweaking that configured threshold in the performance health-check to match the actual performance of the ISP link.