Technical Tip: Troubleshooting IPsec VPN tunnel errors with large size packets

[aionescu]

Description
This article describes techniques on how to identify and troubleshoot VPN tunnel errors due to large size packets.

To confirm errors are increasing on IPsec VPN interface(s), periodically issue one of the below commands:

A) fnsysctl ifconfig <Phase 1 name>

      RX packets:0 errors:0 dropped:0 overruns:0 frame:0
      TX packets:337 errors:1 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:0

B)

#diagnose netlink interface list <Phase 1 name>
stat: rxp=15172 txp=26662 rxb=2994702 txb=3515847 rxe=0 txe=0 rxd=0 txd=0 mc=6529 collision=0

Counters which are marked as red needed to observed.

Solution
There are cases where applications are sending large packets with the don’t fragment (DF) field set to 1.
This can be checked if traffic is captured and analyzed via wireshark by expanding the Internet Protocol field, output, like here below, can show up:

Internet Protocol Version 4, Src: 10.176.2.116, Dst: 172.16.23.171
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 52
    Identification: 0x6758 (26456)
    Flags: 0x4000, Don't fragment
        0... .... .... .... = Reserved bit: Not set
        .1.. .... .... .... = Don't fragment: Set
        ..0. .... .... .... = More fragments: Not set
        ...0 0000 0000 0000 = Fragment offset: 0
    Time to live: 61
    Protocol: TCP (6)
    Header checksum: 0x058d [validation disabled]
    [Header checksum status: Unverified]
    Source: 10.176.2.116
    Destination: 172.16.23.171
Transmission Control Protocol, Src Port: 33274, Dst Port: 13496, Seq: 1, Ack: 1, Len: 0

FortiOS constructs the MTU to the remote peer based on PMTU calculations.
MTU of an IPsec interface is not configurable. The final and most accurate calculation is only done when traffic is starting to traverse the tunnel interface.
The MTU value can be seen via the command:

#diagnose vpn tunnel list name <Phase 1 name>

proxyid_num=1 child_num=0 refcnt=14 ilast=10 olast=10 ad=/0
stat: rxp=1 txp=2034 rxb=96 txb=135331
dpd: mode=on-demand on=1 idle=20000ms retry=3 count=0 seqno=4647
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=to10.189.0.31 proto=0 sa=1 ref=3 serial=5
  src: 0:0.0.0.0/0.0.0.0:0
  dst: 0:0.0.0.0/0.0.0.0:0
  SA:  ref=6 options=10226 type=00 soft=0 mtu=1446 expire=10179/0B replaywin=2048
       seqno=286 esn=0 replaywin_lastseq=00000000 itn=0 qat=0
  life: type=01 bytes=0/0 timeout=42898/43200
  dec: spi=4e0e063f esp=3des key=24 babc8354887c812197f9ae63425dff608a5bd7b3b6201084
       ah=md5 key=16 1d4cba000f30e5b212cfe32aee0c818a
  enc: spi=8bf3aadf esp=3des key=24 7b603fc045ad8a8545f00c6020ba6bb7804e412f96337e8a
       ah=md5 key=16 78430d90d026aed823ec021f944fe6f5
  dec:pkts/bytes=0/0, enc:pkts/bytes=645/81143
  npu_flag=03 npu_rgwy=10.189.0.31 npu_lgwy=10.189.0.182 npu_selid=4 dec_npuid=2 enc_npuid=2
run_tally=1

MTU can be adjusted via two ways:

1) Adjusting the MTU of the physical interface where the IPsec tunnel is bound to.
This method will not only affect the VPN traffic but all traffic which is traversing the physical interface as well.

2) Changing the encryption algorithms.
Stronger encryption algorithms equals to lower MTU values.

For example, the FortiGate sets an IPsec tunnel Maximum Transmission Unit (MTU) of:

1446 for 3des-sha1,

1438 for aes256-sha256, aes192-sha256, aes128-sha1, aes128-sha256

1422 for aes256-sha384, aes256-sha512, aes192-sha384

In case of  NAT-T

1438 for 3des-sha1

1422 for aes256-sha256, aes256-sha384, aes192-sha256, aes192-sha384, aes128-sha1, aes128-sha256

1406 for aes256-sha512

3) Adjusting the MTU of the ISPEC VPN interface using the command below (setting available from FortiOS 6.4).

 

# config system interface

    edit <Phase 1 name>

        set mtu-override enable 

        set mtu 1400

    end

end

If the packet size is greater than the tunnel’s MTU, DF-bit is honored and the IPsec engine drops the packet and the error counters will be increased.
This behavior can be changed with the command:

#config system global

set honor-df [enable|disable]  // Default=enable

If the honor-df field is set to disable, then FortiOS will ignore the packet’s DF-bit settings by encapsulating and encrypting it.
Since the encapsulated packet size will be large enough to exceed the allowed MTU, FortiOS will perform post IPsec fragmentation.