Troubleshooting Tip: Troubleshooting real server health check monitoring

[fwilliams]

Description	This article describes the troubleshooting issue when having a virtual server configured on FortiGate it is pointing to real servers monitored and wanted to respond appropriately if any of the real servers stopped responding to health check monitoring.
Scope	FortiOS.
Solution	Topology: If the monitoring of the real server/s stopped working or the application on the real server suddenly became unreachable, one of the first things to check should be the health check monitoring, to see if the server is ALIVE or DEAD, as FortiGate's VIP does not forward traffic to dead servers. Below could be the likely cause: Network issue between the real server (application server on your network) and the FortiGate. To isolate the network reachability issue as the cause, it is possible to ping the real server’s IP address. If it responds then it is very likely that it is not the cause. Service issue: services are represented by TCP/UDP ports. If the real server/s is a mail server, for example, TCP 25 is likely going to be the TCP port the real server is listening on. It is possible to verify this by 'execute telnet x.x.x.x 25' from the FortiGate. If this show connected then the service is NOT the cause and the server is accepting a connection on this port. Find out if configuration change(s) or upgrades were made recently. If yes, review the configuration change made. For example, changing the VIP object name on the FortiGate can cause issues; especially if the name is a long one. The VIP object's name might not be stored correctly in the KERNEL. To verify this compare the VIP object name in the configuration with the name stored in the kernel by using cmd: show firewall vip <vip_obj_name>   <- To check name in config. diagnose firewall vip realserver list   < To check name stored in the kernel. In this example, the VIP object name in the configuration is 'Long-VIP-object-naming-in-troubleshooting-real-server-health-monitoring.net'. But the one stored in the kernel is 'Long-VIP-object-naming-in-troubleshooting-real-server-health-m'. If this happens, the real server health check monitoring will stop working and the stats will show zeros on all parameters (failed or successful) as seen below. Also, make sure to check the ldb-monitor configuration, and ensure the src-ip, if set, is an IP address reachable from the real servers. If not set, verify what the source IP for the health check monitoring packets is. It is possible to find this information in the ipldbd debug. If the source IP for the health check monitoring is wrong or it is not what is desired, change it under 'config firewall ldb-monitor'. Use the below commands to check the statistics of the health check monitoring packets sent to the real server and how many failed or succeed. It is possible to submit the debug logs printout to TAC if still unable to find what the issue is, and need assistance. diagnose firewall vip realserver list <- Show real server status (alive or dead). <\ol> diagnose firewall vip realserver healthcheck stats <- One-time print health check stats. diagnose debug enable diagnose debug application ipldbd 255 <- Daemon debug. Example of ipldbd debug with failed health check monitor because of WRONG src-ip (10.251.2.100). Example of ipldbd debug with successful health check monitor with the right src-ip (10.10.10.1). Example of stats with working health check monitor.