Troubleshooting Tip: TLS Server Hello packet failed to pass through
| Description | This article explains a quick way to identify if the TLS connection is failing due to a larger server hello packet size. |
| Scope | FortiGate, FortiGate-VM. |
| Solution | In the case that the TLS handshake fails, and the forward logs are shown as server reset or a client reset, Wireshark captures can be utilized.
Run a packet capture for the destination IP and verify the below.
Check for the TLS stream and see if there is a Retransmission just after the Client Hello or Server Hello packets.
In the above captures, the Server Hello packet did not get through the line. The same capture can give more insight if captures are also taken on the end client as well, which will confirm if the actual Server Hello packet was not received on the test machine.
If there is an FQDN in question or it is difficult to run the capture for a single destination IP, filter the URL/application in question using below filter: The frame contains 'URL', replace URL with SNI seen in the Client Hello packet.
After applying the above filter, follow the TCP stream.
If similar behavior is seen, then try to reduce the TCP MSS size on the policy level.
config firewall policy edit <policy id> set tcp-mss-sender <mss value> set tcp-mss-receiver <mss value> next end
If there is any L3 gateway involved in the traffic flow, in such cases, try to collect the Wireshark captures from the end user as well to compare the packet that was received through (larger size TLS packets).
Related articles: Technical Tip: Identifying MSS issues while accessing a website with IPS debug Technical Tip: Setting TCP MSS value Technical Tip: Behavior of TCP-MSS setting under system interface |
