Fragmentation of ESP packets - truncated-ip

Report Inappropriate Content · ‎08-06-2008

Hello everyone, I am experincing a lot of fragmentation on all my VPNs. I discovered this when we set up a new VPN over a new MPLS line and thought it was a problem in the MPLS - but that is fine. It also appears to happen on the VPNs that go over the Internet. I tried setting the tcp-mss and MTU to lower values, but this did not help. Now I heard that it may be possible disallow the fragmentation of packets. Do you know if this is possible or if there is anything else I can do? Here is what I see between my VPN peers (FGT400a to FGT50a/60b):

9.164151 10.24.1.1 -> 10.24.10.1:  ip-proto-50 76
 9.164575 10.24.1.1 -> 10.24.10.1:  ip-proto-50 76
 9.173824 10.24.1.1 -> 10.24.10.1:  ip-proto-50 156
 9.174828 10.24.10.1 -> 10.24.1.1:  ip-proto-50 364
 9.174828 truncated-ip - 20 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 364
 9.183676 10.24.10.1 -> 10.24.1.1:  ip-proto-50 308
 9.183676 truncated-ip - 21 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 308
 9.183970 10.24.1.1 -> 10.24.10.1:  ip-proto-50 76
 9.184073 10.24.10.1 -> 10.24.1.1:  ip-proto-50 92
 9.184073 truncated-ip - 20 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 92
 9.265988 10.24.1.1 -> 10.24.10.1:  ip-proto-50 84
 9.278419 10.24.10.1 -> 10.24.1.1:  ip-proto-50 108
 9.278419 truncated-ip - 19 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 108
 9.287059 10.24.10.1 -> 10.24.1.1:  ip-proto-50 244
 9.287059 truncated-ip - 18 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 244
 9.287436 10.24.1.1 -> 10.24.10.1:  ip-proto-50 76
 9.295359 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1460
 9.295359 truncated-ip - 19 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1460
 9.296935 10.24.10.1 -> 10.24.1.1:  ip-proto-50 460
 9.296935 truncated-ip - 16 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 460
 9.297291 10.24.1.1 -> 10.24.10.1:  ip-proto-50 76
 9.303517 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1468
 9.303517 truncated-ip - 16 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1468
 9.309656 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1468
 9.309656 truncated-ip - 16 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1468
 9.310627 10.24.1.1 -> 10.24.10.1:  ip-proto-50 76
 9.312327 10.24.10.1 -> 10.24.1.1:  ip-proto-50 724
 9.312327 truncated-ip - 15 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 724
 9.318821 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1468
 9.318821 truncated-ip - 16 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1468
 9.319606 10.24.1.1 -> 10.24.10.1:  ip-proto-50 76
 9.325128 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1468
 9.325128 truncated-ip - 16 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1468
 9.328197 10.24.10.1 -> 10.24.1.1:  ip-proto-50 836
 9.328197 truncated-ip - 18 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 836
 9.328880 10.24.1.1 -> 10.24.10.1:  ip-proto-50 76
 9.334591 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1468
 9.334591 truncated-ip - 16 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1468
 9.340721 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1468
 9.340721 truncated-ip - 16 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1468
 9.340923 10.24.10.1 -> 10.24.1.1:  ip-proto-50 172
 9.340923 truncated-ip - 16 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 172
 9.341633 10.24.1.1 -> 10.24.10.1:  ip-proto-50 76
 9.347794 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1468
 9.347794 truncated-ip - 16 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1468
 9.348879 10.24.1.1 -> 10.24.10.1:  ip-proto-50 76
 9.351979 10.24.10.1 -> 10.24.1.1:  ip-proto-50 988
 9.351979 truncated-ip - 18 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 988
 9.358363 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1468
 9.358363 truncated-ip - 16 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1468
 9.359311 10.24.1.1 -> 10.24.10.1:  ip-proto-50 76
 9.364560 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1468
 9.364560 truncated-ip - 16 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1468
 9.369774 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1268
 9.369774 truncated-ip - 16 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1268
 9.370619 10.24.1.1 -> 10.24.10.1:  ip-proto-50 76
 9.374457 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1140
 9.374457 truncated-ip - 14 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1140
 9.377278 10.24.10.1 -> 10.24.1.1:  ip-proto-50 724
 9.377278 truncated-ip - 21 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 724
 9.377864 10.24.1.1 -> 10.24.10.1:  ip-proto-50 76
 9.384344 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1420
 9.384344 truncated-ip - 17 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1420
 9.481649 10.24.10.1 -> 10.24.1.1:  ip-proto-50 116
 9.481649 truncated-ip - 20 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 116
 9.482494 10.24.1.1 -> 10.24.10.1:  ip-proto-50 76
 11.638381 10.24.10.1 -> 10.24.1.1:  ip-proto-50 92
 11.638381 truncated-ip - 16 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 92
 11.638849 10.24.1.1 -> 10.24.10.1:  ip-proto-50 92
 15.635853 10.24.1.1 -> 10.24.10.1:  ip-proto-50 92
 15.669933 10.24.10.1 -> 10.24.1.1:  ip-proto-50 100
 15.669933 truncated-ip - 14 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 100
 15.731477 10.24.1.1 -> 10.24.10.1:  ip-proto-50 132
 16.044901 10.24.10.1 -> 10.24.1.1:  ip-proto-50 76
 16.044901 truncated-ip - 20 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 76
 16.108037 10.24.1.1 -> 10.24.10.1:  ip-proto-50 148
 16.154262 10.24.10.1 -> 10.24.1.1:  ip-proto-50 76
 16.154262 truncated-ip - 20 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 76
 18.405066 10.24.10.1 -> 10.24.1.1:  ip-proto-50 92
 18.405066 truncated-ip - 16 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 92
 18.405469 10.24.1.1 -> 10.24.10.1:  ip-proto-50 92
 22.615196 10.24.10.1 -> 10.24.1.1:  ip-proto-50 84
 22.615196 truncated-ip - 20 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 84
 22.615595 10.24.1.1 -> 10.24.10.1:  ip-proto-50 84
 22.634223 10.24.10.1 -> 10.24.1.1:  ip-proto-50 76
 22.634223 truncated-ip - 20 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 76
 22.635951 10.24.10.1 -> 10.24.1.1:  ip-proto-50 356
 22.635951 truncated-ip - 21 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 356
 22.642641 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1468
 22.642641 truncated-ip - 16 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1468
 22.643239 10.24.1.1 -> 10.24.10.1:  ip-proto-50 76
 22.668604 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1468
 22.668604 truncated-ip - 16 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1468
 22.674708 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1468
 22.674708 truncated-ip - 16 bytes missing! 10.24.10.1 -> 10.24.1.1:  ip-proto-50 1468

Thanks for reading! stephan

rwpatterson · ‎08-06-2008

Run a ping across the connection with the ' -f' flag (do not truncate) and the ' -l' flag (packet length in bytes, that' s a lower case " L" ). Find out what the largest packet that can be sent is, and adjust your tcp-mss accordingly. I found mine to be 142x bytes before fragmenting, so I set my tcp-mss to 1400 across all tunnels. This should be done on both sides of the IPSec VPN tunnel. This has sped up the remote sites greatly since they no longer have to break down and reassemble each packet across the WAN. Use the below command:

> ping -f -l 1400 192.168.x.x

Good luck

Bob - self proclaimed posting junkie!
See my Fortigate related scripts at: http://fortigate.camerabob.com

Report Inappropriate Content · ‎08-06-2008

Hi Bob, thanks for your help. I understand what you mean, but I' m not sure if that can help me. I tried pinging with different lenghts, now I tried -f as well. Does not seem to have an effect (with -l 1400, 1200, 1000): 34.436494 10.24.1.1 -> 10.24.2.1: ip-proto-50 1460 34.456613 10.24.2.1 -> 10.24.1.1: ip-proto-50 1460 34.456613 truncated-ip - 16 bytes missing! 10.24.2.1 -> 10.24.1.1: ip-proto-50 1460 35.437245 10.24.1.1 -> 10.24.2.1: ip-proto-50 1460 35.457505 10.24.2.1 -> 10.24.1.1: ip-proto-50 1460 35.457505 truncated-ip - 16 bytes missing! 10.24.2.1 -> 10.24.1.1: ip-proto-50 1460 36.437759 10.24.1.1 -> 10.24.2.1: ip-proto-50 1460 36.458004 10.24.2.1 -> 10.24.1.1: ip-proto-50 1460 36.458004 truncated-ip - 16 bytes missing! 10.24.2.1 -> 10.24.1.1: ip-proto-50 1460 41.446148 10.24.2.1.500 -> 10.24.1.1.500: udp 92 41.446426 10.24.1.1.500 -> 10.24.2.1.500: udp 92 43.476714 10.24.1.1 -> 10.24.2.1: ip-proto-50 1260 43.494879 10.24.2.1 -> 10.24.1.1: ip-proto-50 1260 43.494879 truncated-ip - 16 bytes missing! 10.24.2.1 -> 10.24.1.1: ip-proto-50 1260 50.483658 10.24.2.1.500 -> 10.24.1.1.500: udp 92 50.483927 10.24.1.1.500 -> 10.24.2.1.500: udp 92 51.106043 10.24.1.1 -> 10.24.2.1: ip-proto-50 1060 51.122140 10.24.2.1 -> 10.24.1.1: ip-proto-50 1060 51.122140 truncated-ip - 16 bytes missing! 10.24.2.1 -> 10.24.1.1: ip-proto-50 1060 When I ping from Fortigate to Fortigate (non-VPN), it goes unfragmented when I set the data-size to 1472, which is correct: 1472 = payload + 20 = IP header + 8 = ICMP Header = 1500 So I know the MTU size is correct. When I use TCP (HTTPS) through the WAN link I do not see any fragmentation messages. Seems like only ESP is affected. Any thoughts? Thanks stephan

rwpatterson · ‎08-06-2008

What versions of firmware are you running?

Bob - self proclaimed posting junkie!
See my Fortigate related scripts at: http://fortigate.camerabob.com

Report Inappropriate Content · ‎08-06-2008

Fortigate 400A: 3.00-b0662(MR6 Patch 1) Fortigate-60B No1: 3.00-b0662(MR6 Patch 1) Fortigate-60B No2: 3.00-b0726(MR7) Thought it might be a problem with the 400a, but it happens between the two 60Bs as well :(

rwpatterson · ‎08-06-2008

From the CLI, run the following command:

diagnose hardware deviceinfo nic interface_name

If you have errors, run the command additional times and see if the count grows. You may have an autonegotiation issue as I did, or a duplex mismatch.

Bob - self proclaimed posting junkie!
See my Fortigate related scripts at: http://fortigate.camerabob.com

Report Inappropriate Content · ‎08-07-2008

Hi Bob, I get this on the Fortigate 400:

 FG400A-2 # diagnose hardware deviceinfo nic port4
 Description               Intel(R) PRO/100 M Desktop Adapter
 Driver_Name               e100
 Driver_Version            2.1.29
 PCI_Vendor                0x8086
 PCI_Device_ID             0x1229
 PCI_Subsystem_Vendor      0x8086
 PCI_Subsystem_ID          0x0070
 PCI_Revision_ID           0x0010
 PCI_Bus                   3
 PCI_Slot                  6
 IRQ                       20
 System_Device_Name        port4
 Current_HWaddr            00:09:0F:09:00:03
 Permanent_HWaddr          00:09:0F:84:76:ED
 Part_Number               ffffff-0ff
 
 Link                      up
 Speed                     100
 Duplex                    full
 FlowControl               receive/transmit
 State                     up
 
 Rx_Packets                2877827
 Tx_Packets                1022152
 Rx_Bytes                  1619676618
 Tx_Bytes                  452862314
 Rx_Errors                 0
 Tx_Errors                 10
 Rx_Dropped                0
 Tx_Dropped                0
 Multicast                 N/A
 Collisions                0
 Rx_Length_Errors          0
 Rx_Over_Errors            0
 Rx_CRC_Errors             0
 Rx_Frame_Errors           0
 Rx_FIFO_Errors            0
 Rx_Missed_Errors          0
 Tx_Aborted_Errors         0
 Tx_Carrier_Errors         10
 Tx_FIFO_Errors            0
 Tx_Heartbeat_Errors       0
 Tx_Window_Errors          0
 
 Rx_TCP_Checksum_Good      0
 Rx_TCP_Checksum_Bad       0
 Tx_TCP_Checksum_Good      0
 Tx_TCP_Checksum_Bad       0
 
 Tx_Single_Collision_Frames 0
 Tx_Multi_Collision_Frames 0
 Tx_Deferred               0
 Rx_Symbol_Errors          0
 
 Tx_Pause_Frames           0
 Rx_Pause_Frames           0
 Rx_Control_Unknown_Opcodes 0
 
 Tx_TCO_Packets            0
 Rx_TCO_Packets            0
 
 Rx_Interrupt_Packets      0
 Rx_Polling_Packets        2879708
 Polling_Interrupt_Switch  0

Rx/TxBytes and Packtes increase, other than that, only Rx_Polling_Packets increase. On the 60Bs:

 # diagnose hardware deviceinfo nic wan1
 Description             sundance Ethernet driver1.01+LK1.21
 chip_id                 6                                                                   
 IRQ                     5                                                                   
 System_Device_Name      wan1
 Current_HWaddr          00:09:0f:79:0a:ae
 Permanent_HWaddr        00:09:0f:79:0a:ae
 State                   up                                                                  
 Link                    up                                                                  
 Speed                   100                                                                 
 Duplex                  full                                                                
 Rx_Packets              621178                                                              
 Tx_Packets              963582                                                              
 Rx_Bytes                112144056                                                           
 Tx_Bytes                479493981                                                           
 Collisions              0                                                                   
 Rx_Missed_Errors        0
 Tx_Carrier_Errors       0
 
 and
 
 # diagnose hardware deviceinfo nic wan1
 Description             sundance Ethernet driver1.01+LK1.21
 chip_id                 6
 IRQ                     5
 System_Device_Name      wan1
 Current_HWaddr          00:09:0f:79:05:62
 Permanent_HWaddr        00:09:0f:79:05:62
 State                   up
 Link                    up
 Speed                   100
 Duplex                  full
 Rx_Packets              478361
 Tx_Packets              601930
 Rx_Bytes                232205448
 Tx_Bytes                510926723
 Collisions              0
 Rx_Missed_Errors        0
 Tx_Carrier_Errors       0

100/Full is correct everywhere... Thanks stephan

rwpatterson · ‎08-07-2008

Also check the inside port(s) the internal device is on...

Bob - self proclaimed posting junkie!
See my Fortigate related scripts at: http://fortigate.camerabob.com

Report Inappropriate Content · ‎08-07-2008

On the LAN interface of the FG400 I see these: Rx_CSum_Offload_Good 1197420231 rising at about 400/second Rx_CSum_Offload_Errors 305 Errors not rising. I see no errors on the internal interfaces of the FG60s or at the connected switches. A consultant from a Network specialist here told me they have it on ALL their Fortigate VPNs and never checked whether this could be a problem. They wondered though, but never found the time to analyze it... I promised to keep him up to date ;( No response to my ticket from yesterday yet...

Report Inappropriate Content · ‎08-12-2008

After sending some traces and some discussion with the Fortinet Support they came to this conclusion:

The truncated-ip message is an expected behavior. What happens is that when Fortigate gets packets through the VPN it tries to match the packet header as a normal packet but it does not match thats why it shows it as truncated packet. Its normal for VPN traffic and it does not create any performance problem on the network or on the unit.

You will not see truncated-ip messages when sniffing on interface any - this is something like when do a trace on a LAN interface with Wireshark and get currupted TCP checksums. Everything fine, just the sniffer getting something wrong :-) Thanks.

Fragmentation of ESP packets - truncated-ip

Nominate a Forum Post for Knowledge Article Creation