Description
This article describes how, in certain circumstances, a FortiGate deployment may experience higher packet loss than normal and some common reasons for this behavior. There are also recommendations on how to resolve common issues or test hardware for possible problems.
Scope
FortiGate.
Solution
Several factors can cause packet loss on the FortiGate:
- Incorrect speed settings on the interface.
Check the speed settings on each interface from the GUI by moving the mouse over the interface on System -> Status -> Unit Operation or by running the following CLI command:
dia hard device nic <interface name>
To check if there are errors in the interface, use the following command:
fnsysctl ifconfig port1
port1 Link encap:Ethernet HWaddr D4:76:A0:1C:6D:B4
inet addr:192.168.10.1 Bcast:192.168.10.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:10608 errors:0 dropped:0 overruns:0 frame:0
TX packets:5437 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2232859 (2.1 MB) TX bytes:684968 (668.9 KB)
Users will be looking for a speed of 10half. This usually means that the FortiGate was not able to negotiate the speed correctly with the device on the other side.
To set the speed manually, use the commands:
config system interface
edit <interface name>
set speed 100full
end
Warning: Some vendors will turn off the interface if auto-negotiate is turned off on the FortiGate. Make sure not to be connected through the same link being changed or connection to the FortiGate may be lost.
To check if there are errors or drops in an interface, use the above commands.
-
High bandwidth usage.
To generate bandwidth reports, make sure to have enabled logging on firewall policies. This is done by going to Firewall -> Policy and editing the policies. Enable logging by enabling 'log allowed traffic'.
On a FortiAnalyzer, go to Report -> Config -> Layout -> Create New -> Add charts as needed. Most users will need Traffic Volume by Direction, Top Services by Volume, and Top Sources by Volume.
In Report -> Schedule -> Create New -> use the layout that was just created and select the devices (that is: FortiGates) on which to run the report. Select OK. Schedule the report or run it on demand using the 'Run now' icon on the Report -> Schedule page.
-
Hardware issues.
Finally, the problem can be caused by a hardware problem. Administrators can run a hardware test to check for any hardware problem, or problems with interfaces.
The hardware test is executed as follows:
- Go to the Support Portal at support.fortinet.com, login, and take the Download -> HQIP Images option. The related article 'RMA Note: HQIP - Hardware Quick Inspection Package' provides information about running HQIP tests. An outline description is also presented below.
- Download the HQIP diagnostics firmware Image for the FortiGate unit, and save it in the root directory of a TFTP server.
- Connect the PC Ethernet port to the internal interface of the FortiGate unit using a cross-over cable.
- Connect a PC serial port to the console port of the unit and start a terminal client application program such as Hyper terminal.
Set the terminal client for serial communications as follows:
Baud rate: 9600
Data: 8
Parity: none
Stop: 1
Flow Control: none
Set the terminal to capture output from the console and save it in a text file. After completing this test successfully (or unsuccessfully), attach this text file to a Fortinet Support ticket already opened to resolve this issue.
- Power on the FortiGate unit. Interrupt the boot process when the 'press any key to display configuration' message is displayed on the console screen.
- Select G to get the firmware from the selection menu:
[G]: Get firmware image from TFTP server.
[F]: Format boot device.
[Q]: Quit menu and continue to boot with default firmware.
[H]: Display this list of options.
- Enter the IP address of the TFTP computer (both IP addresses below have to be in the same subnet).
Enter TFTP server address [192.168.1.168]: Use the current PC IP address, or configure the PC to a static IP address of 192.168.1.168.
- Enter local address [192.168.1.188]: Use an IP address other than the one above, but in the same subnet.
- Enter the HQIP file name.
- When prompted with the choice to save as Default, save as Backup, or Run image without saving, Select 'R' to run without saving.
- Attach output to the ticket.
-
Traffic Shapers.
If a traffic shaper was applied, check the session list for possible drops.
Run 'diagnose system session list' to see the session list details.
Endeavour-kvm96 # diagnose system session list
session info: proto=6 proto_state=11 duration=30 expire=3599 timeout=3600 flags=00000000 socktype=0 sockport=0 av_idx=0 use=4
origin-shaper=high-priority prio=2 guarantee 0Bps max 131072000Bps traffic 186Bps drops 0B <-------
reply-shaper=high-priority prio=2 guarantee 0Bps max 131072000Bps traffic 186Bps drops 0B <------
per_ip_shaper=
class_id=0 shaping_policy_id=1 ha_id=0 policy_dir=0 tunnel=/ vlan_cos=0/255
state=log may_dirty ndr os rs f00
statistic(bytes/packets/allow_err): org=1467/12/1 reply=1262/8/1 tuples=3
tx speed(Bps/kbps): 47/0 rx speed(Bps/kbps): 40/0
orgin->sink: org pre->post, reply pre->post dev=5->3/3->5 gwy=10.9.15.254/0.0.0.0
hook=post dir=org act=snat 192.168.1.2:56525->34.107.221.82:80(10.9.12.64:56525)
hook=pre dir=reply act=dnat 34.107.221.82:80->10.9.12.64:56525(192.168.1.2:56525)
hook=post dir=reply act=noop 34.107.221.82:80->192.168.1.2:56525(0.0.0.0:0)
pos/(before,after) 0/(0,0), 0/(0,0)
misc=0 policy_id=1 pol_uuid_idx=15747 auth_info=0 chk_client_info=0 vd=0
serial=0010ca5c tos=ff/ff app_list=0 app=0 url_cat=0
rpdb_link_id=00000000 ngfwid=n/a
npu_state=0x001108
no_ofld_reason: redir-to-ips denied-by-nturbo
FortiGate may drop packets due to high memory or CPU usage.
Run 'get system performance status' to find the CPU and memory usage.
Endeavour-kvm96 # get sys performance status
CPU states: 0% user 1% system 0% nice 99% idle 0% iowait 0% irq 0% softirq <---
CPU0 states: 0% user 1% system 0% nice 99% idle 0% iowait 0% irq 0% softirq <---
Memory: 2040052k total, 966408k used (47.4%), 663468k free (32.5%), 410176k freeable (20.1%) <---
Average network usage: 38 / 3 kbps in 1 minute, 41 / 5 kbps in 10 minutes, 40 / 5 kbps in 30 minutes
Maximal network usage: 56 / 11 kbps in 1 minute, 409 / 110 kbps in 10 minutes, 409 / 144 kbps in 30 minutes
Average sessions: 43 sessions in 1 minute, 24 sessions in 10 minutes, 24 sessions in 30 minutes
Maximal sessions: 47 sessions in 1 minute, 48 sessions in 10 minutes, 51 sessions in 30 minutes
Average session setup rate: 0 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutes
Maximal session setup rate: 0 sessions per second in last 1 minute, 16 sessions per second in last 10 minutes, 23 sessions per second in last 30 minutes
Average NPU sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes
Maximal NPU sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes
Virus caught: 0 total in 1 minute
IPS attacks blocked: 0 total in 1 minute
Uptime: 4 days, 8 hours, 5 minutes
Further troubleshooting, to identify whether the issue is with the FortiGate or not, requires running the packet capture on the ingress interface and egress interface of the firewall for the destination IP. Initiate the ICMP packets from the source machine to the destination and observe the packet loss.
