Troubleshooting Tip: Troubleshooting network issues when the FortiGate is hosted on a KVM hypervisor

[johnathan]

Description	This article describes techniques used to troubleshoot network issues when there is a FortiGate hosted on a KVM hypervisor and there may be an issue with said hypervisor.
Scope	FortiGate VM, hosted on KVM
Solution	When troubleshooting network issues with a FortiGate on a hypervisor, it is useful to know how to trace traffic after it leaves the FortiGate. This may give a better indication as to where the issue lies. For example, if  the HA cluster is intermittently losing heartbeats between the units it would be helpful to know if there are packets whom are leaving the FortiGate and are not seen entering the hypervisor, or if packets are entering the hypervisor but not being forwarded to the FortiGate.  Depending on the hypervisor, the tools used to do this can be different. This article only covers KVM, there should be vendor documentation for whichever hypervisor the FortiGate was deployed on. It's important to determine which interface on the FortiGate is correlated with the attached interface inside the hypervisor. The port number depends on the order it was added into the FortiGate, which is the same as the order it was configured in. 'port1' would be the first interface in the configuration, 'port2' would be the second, etc. This article goes over how networking in general works when using KVM: Virtual Networking - LibVirt.org wiki. The important thing to take away is that VMs are assigned to a virtual switch (virbrX), inside of that virtual  switch is usually a way to the physical port on the hypervisor. Depending on how the virtual switch is set up, this may be routed (each VM has an IP in the network on the physical port), or NAT'ted (shares the IP of the physical port, different subnet on the inside of the switch). It's possible to choose to have a virtual switch with no way to a physical port, this will be isolated from the outside. Generally as long as it is possible to see the packets enter the virtual switch (and they have the correct destination MAC address), the FortiGate had done it's job. It is up to the hypervisor to forward packets where they need to go once it reaches this stage.  To list the interfaces assigned to  a VM, run 'virsh domiflist <VM Name>': This will give the virtual network name ('default' in this case!). To see the virtual switch assigned to that virtual network, run 'virsh net-info <Network Name>':   To sniff packets in Linux, the command is 'tcpdump'. This command is very very similar to the 'di sni pack' command built into FortiOS, the filter syntax is the same. This is the following command used in the screenshot: tcpdump -nn -i virbr0 host 1.1.1.1 '-nn' makes the output cleaner. '-i' specifies the interface to listen on. Here is an example of this command, with some output: Setting the interface to 'any' will show the packet exit the physical port: