Troubleshooting Tip: Initial troubleshooting commands for Virtual Servers

Verification and debug.

Check the status of the real servers:

diagnose firewall vip realserver

Any of the following options can be supplied:

list: create a list.

up: change the address to 'up'.

down: change the address to 'down'.

healthcheck: perform a server health check.

clear: clear the firewall VIP, VIP6 real server statistics.

Information about Virtual Servers can also be viewed in the GUI under Monitoring -> Load Balance Monitor.

The health check configuration can be verified using the following CLI command:

   config firewall ldb-monitor

If the health check is configured as an HTTPS check, verify the source IP, port number, timeout values, and website redirections, for example: www.example.com/page.

Change the Health Check to ICMP with proper timeout values. 

Once the Load Balance monitor shows UP, then the website should start working.

For general stats:

get test ipldbd 2 (where ipldbd is the load balance daemon).

Example output:

num of vf=1
--------dump ipldb vf=0----------
num of vips=1
num of registered monitor types=6
num of ping monitors=0
num of ping monitors=3
num of tcp monitors=0
num of http monitors=0
num of https monitors=0
num of dns monitors=0

The following command displays VIP filters. This is useful on FortiGates with many VIPs:

diagnose firewall vip virtual-server filter <option>

list: display the current filter.

clear: erase the current filter.

name: specify a VIP name to filter by.

src: specify a source address range to filter by.

dst: specify a destination address range to filter by.

src-port: specify a source port range to filter by.

dst-port: specify a destination port range to filter by.

vd: specify the index of a virtual domain. -1 matches all.

negate: negate the specified filter parameter.

Use the following command to view virtual server stats:

diagnose firewall vip virtual-server stats <option>

list            List all statistics.
clear           Clear all statistics.
http            HTTP statistics.
ssl             SSL statistics.
crypto-clear    Clear SSL crypto statistics.
operational     Operational info and statistics.
summary         Summary statistics.

The best verification is a packet sniffer. In the sniffer, it is possible to view how packets are being forwarded based on the current load-balancing method. 

diagnose sniffer packet <interface/any> ‘<filter>’ <verbose> <count>
<Timestamp format>

To stop the sniffer, press Ctrl+C on the keyboard.

It is possible to change the format above to sniff for the specific traffic being forwarded towards the servers.

The following commands' output can be helpful to start initial troubleshooting and understand the Virtual server issue.

fnsysctl date
diagnose firewall vip virtual-server real-server list
diagnose firewall vip virtual-server server
diagnose firewall vip virtual-server stats list
diagnose firewall vip virtual-server session list
diagnose firewall vip realserver healthcheck stats show
diagnose firewall vip realserver list

get router info routing-table all
get router info routing-table database
get router info kernel
diagnose firewall proute list
diagnose ip rtcache list

Find below the live debug for troubleshooting when working with FortiGate real servers 

 diagnose debug disable

 diagnose debug reset

 diagnose debug application ipldbd -1

 diagnose debug enable

Run the debug, replicate the problem, and stop the capture with:

diagnose debug disable

See this document Load balancing diagnose commands for more information about load balancing diagnosis commands:

Related documents:

Virtual server load balance

Technical Tip: Configure virtual server

Troubleshooting Tip: How to fix an issue with Virtual Servers