Technical Tip: Using link-monitor debugging to troubleshoot SD-WAN http based link-monitor setups

[PabloSaco]

Description	This article describes how to use link-monitor debugging to troubleshoot HTTP-based link-monitor issues.
Scope	FortiOS, FortiGate.
Solution	The following link-monitor configuration will be used for this article: config health-check     edit "HTTP_Server1"         set server "192.168.100.101"         set protocol http         set http-match "fortinet"         set members 3 4 5     next end The FortiGate will attempt to reach the configured server through HTTP instead of using the standard ICMP echo requests. When looking at the health check's status, the following is displayed: Fortigate # diagnose sys sdwan health-check status HTTP_Server1 Health Check (HTTP_Server1): Seq(3 Branch1_VPN1): state (dead), packet-loss (100.000%) sla_map=0x0 Seq(4 Branch1_VPN2): state (dead), packet-loss (100.000%) sla_map=0x0 Seq(5 port4): state (dead), packet-loss (100.000%) sla_map=0x0 FortiGate reports a 100% packet loss for the configured members, meaning those links cannot be used to forward traffic from SD-WAN's perspective. When attempting to ping 192.168.100.101, the ping fails: Fortigate # execute ping 192.168.100.101 PING 192.168.100.101 (192.168.100.101): 56 data bytes sendto failed: 22(Invalid argument) sendto failed: 22(Invalid argument) sendto failed: 22(Invalid argument) However, this does not necessarily mean the circuit is down, as ICMP may be filtered by the receiving server. When running a sniffer for HTTP requests, it proves that the server is actually replying to them: Fortigate # diagnose sniffer packet any 'host 192.168.100.101 and port 80' 4 0 a Using Original Sniffing Mode interfaces=[any] filters=[host 192.168.100.101 and port 80] 2024-07-08 14:35:06.491618 Branch1_VPN1 out 10.1.1.1.8007 > 192.168.100.101.80: syn 1082873658 2024-07-08 14:35:06.492145 Branch1_VPN2 out 10.1.2.1.11617 > 192.168.100.101.80: syn 3320109207 2024-07-08 14:35:06.492323 port4 out 10.10.14.1.9643 -> 192.168.100.101.80: syn 2521809051 2024-07-08 14:35:06.493371 Branch1_VPN1 in 192.168.100.101.80 -> 10.1.1.1.8007: syn 2507651606 ack 1082873659 2024-07-08 14:35:06.493444 Branch1_VPN1 out 10.1.1.1.8007 > 192.168.100.101.80: ack 2507651607 To troubleshoot this further, debugging the link-monitor's daemon (lnkmtd) can give greater details. The following commands can be used for that: diagnose debug application link-monitor -1 diagnose debug enable To stop the debug, use the commands below: diagnose debug disable diagnose debug reset The debug shows the following: Fortigate # Inkmtd::http_send_request(551): ---> HTTP monitor HTTP_Server1-3-VIRTUAL_WAN_LINK-3 checking 192.168.100.101:80 ... Inkmtd::http_send_request(551): ---> HTTP monitor HTTP_Server1-4-VIRTUAL_WAN_LINK-4 checking 192.168.100.101:80 ... Inkmtd::http_send_request(551): ---> HTTP monitor HTTP_Server1-5-VIRTUAL_WAN LINK-5 checking 192.168.100.101:80 ... Inkmtd::Inkmt_monitor_check_sock_err(3538): ---> http sock 23 connect succeeded! Inkmtd::http_send_url(407): ---> HTTP send get-url="GET / HTTP/1.1 User-Agent: FortiGate (FortiOS 7.0) Chrome/ Safari/ Host: 192.168.100.101 Keep-Alive: timeout=5 Connection: Keep-Alive Content-Length: 0 - Inkmtd::Inkmt_monitor_check_sock_err(3538): ---> http sock 24 connect succeeded! Inkmtd::http_send_url(407): ---> HTTP send get-url="GET / HTTP/1.1 User-Agent: FortiGate (FortiOS 7.0) Chrome/ Safari/ Host: 192.168.100.101 Keep-Alive: timeout=5 Connection: Keep-Alive Content-Length: 0 - Inkmtd::http_handle_get_response(371): ---> recv failed mon-HTTP_Server1-3-VIRTUAL_WAN_LINK-3 errno-11 Inkmtd::http_handle_get_response(371): ---> recv failed mon-HTTP_Server1-3-VIRTUAL WAN LINK-3 errno=9 Inkmtd::http_handle_get_response(371): ---> recv failed mon-HTTP_Server1-4-VIRTUAL WAN_LINK-4 errno-9 Inkmtd::http_handle_get_response(371): ---> recv failed mon-HTTP_Server1-4-VIRTUAL WAN LINK-4 errno=9 Inkmtd::http_send_request(551): ---> HTTP monitor HTTP_Server1-3-VIRTUAL WAN LINK-3 checking 192.168.100.101:80 ... Inkmtd::http_send_request(551): ---> HTTP monitor HTTP_Server1-4-VIRTUAL_WAN_LINK-4 checking 192.168.100.101:80 ... Inkmed: Inlnt_monitor_check_sock_err(3508). ---> http sock 23 connect succeeded! Inkmtd::http_send_url(407): ---> HTTP send get-url="GET / HTTP/1.1 User-Agent: FortiGate (FortiOS 7.0) Chrome/ Safari/ Host: 192.168.100.101 Keep-Alive: timeout=5 Connection: Keep-Alive The server's response fails to satisfy the set parameters, since the HTTP-Match method is being used in this scenario, which could mean that the expected string 'fortinet' is not included in the server's response. To verify this, the server's response should be checked. There are several ways to do this, including packet captures or running curl to the server from a host. The following is an example pertinent to this scenario: root@server:~# curl 192.168.100.101 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">  <html xmlns="http://www.w3.org/1999/xhtml">  <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> <title>Apache2 Debian Default Page: It works</title> </head> <body> <div class="main_page"> <div class="page_header floating_element"> <img src="/icons/openlogo-75.png" alt="Debian Logo" class="floating_element"/>  <span class="floating_element"> Welcome to the SD-WAN World! </span> </div> </div> </body> </html> The 'fortinet' string is nowhere to be found. To fix this from the FortiGate, the configuration can be adjusted for the FortiGate to expect a string included in the server response: Fortigate # config system sdwan Fortigate (sdwan) # config health-check Fortigate (health-check) # edit HTTP_Server1 Fortigate (HTTP_Server1) # set http-match 'SD-WAN' Fortigate (HTTP_Server1) # end Fortigate (sdwan) # end After this change, SD-WAN members are up, and traffic is being forwarded to them normally: Related document: Link health monitor