Fortinet Forum
The Forums are a place to find answers on a range of Fortinet products from peers and product experts.
seadave
Contributor III

Outlook Clients Connection Errors 6.0.7 SSL-VPN Full Tunnel

Hello-

 

Yesterday we updated our HA 501E pair from 6.0.5 to 6.0.7.  All seemed to be fine, but today, users connected to our SSL VPN tunnel are having Outlook 2016 Connectivity issues.  We have Outlook on-prem and only clients coming in over the SSL VPN full tunnel are having the issue.

 

Outlook sync log keeps generating errors similar to:

14:58:32 Synchronizer Version 16.0.11328 14:58:32 Synchronizing Mailbox 'Joe User' 14:58:32 Synchronizing Hierarchy 14:58:33 Synchronizing server changes in folder 'Inbox' 14:58:33 Downloading from server 'd10db7e0-492a-4521-8c0a-fe49225ad443@company.com' 14:58:34 Downloading from server 'd10db7e0-492a-4521-8c0a-fe49225ad443@company.com' 14:58:34 Error synchronizing folder [size="2"]14:58:34 [80040115-514-80040115-0][/size] 14:58:34 Network problems are preventing connection to Microsoft Exchange. 14:58:34 Microsoft Exchange Information Store 14:58:34 For more information on this failure, click the URL below: 14:58:34 https://www.microsoft.com...0040115-514-80040115-0 14:58:34 Done 14:58:34 Microsoft Exchange offline address book 14:58:34 Download successful

 

Of course the support link goes to a generic MS help page and is unrelated.

 

If we put any of our clients on a mobile hotspot and VPN in, this happens.  If we take them off and reconnect to wifi, no more errors.  So it seems like something is happening to the traffic as it passes through the tunnel.

 

We are using a full tunnel, without split DNS.  Here are the settings:

 

config vpn ssl settings set reqclientcert disable set tlsv1-0 disable set tlsv1-1 disable set tlsv1-2 enable unset banned-cipher set ssl-insert-empty-fragment enable set https-redirect enable set x-content-type-options enable set ssl-client-renegotiation enable set force-two-factor-auth disable [size="2"] set servercert "XXXXXXXX"  (Obscured for post)[/size] set algorithm high set idle-timeout 7200 set auth-timeout 28800 set login-attempt-limit 2 set login-block-time 60 set login-timeout 30 set dtls-hello-timeout 10 set tunnel-ip-pools "SSLVPN_TUNNEL_ADDR1" set tunnel-ipv6-pools "SSLVPN_TUNNEL_IPv6_ADDR1" set dns-suffix "company.local" set dns-server1 XXXXXXXXXXX (Obscured for post) [size="2"] set dns-server2 XXXXXXXXXXX (Obscured for post)[/size] set wins-server1 0.0.0.0 set wins-server2 0.0.0.0 set ipv6-dns-server1 :: set ipv6-dns-server2 :: set ipv6-wins-server1 :: set ipv6-wins-server2 :: set url-obscuration enable set http-compression disable set http-only-cookie enable set port 443 set port-precedence enable set auto-tunnel-static-route enable set header-x-forwarded-for add set source-interface "WAN" set source-address "all" set source-address-negate disable set source-address6 "all" set source-address6-negate disable set default-portal "SSL_VPN_TS" config authentication-rule edit 8 set groups "SSL_VPN_FULL" set portal "SSL_VPN_FULL" set realm '' set client-cert disable set cipher high set auth any next

set dtls-tunnel enable set check-referer enable set http-request-header-timeout 20 set http-request-body-timeout 30 set unsafe-legacy-renegotiation disable

end

 

We attempted to sniff the traffic and the PCAPS do show RSTs occurring.

 

We have used the Default Web Proxy for ever with these connections and tried to use a custom one with RPC over HTTP enabled but it made no change to the behavior.

 

config vpn ssl web portal edit "SSL_VPN_FULL" set tunnel-mode enable [size="2"] set ipv6-tunnel-mode disable [style="background-color: #ffff00;"]<----- This was enabled and we set to disabled, but no luck[/style][/size] set web-mode enable set host-check none set limit-user-logins enable set mac-addr-check disable set os-check disable set forticlient-download disable set ip-mode range set auto-connect disable set keep-alive disable set save-password disable set ip-pools "SSLVPN_TUNNEL_ADDR1" set exclusive-routing disable set service-restriction disable set split-tunneling disable set dns-server1 0.0.0.0 set dns-server2 0.0.0.0 set dns-suffix '' set wins-server1 0.0.0.0 set wins-server2 0.0.0.0 set display-bookmark enable set user-bookmark enable set allow-user-access web ftp smb telnet ssh vnc rdp ping citrix portforward set user-group-bookmark enable

set display-connection-tools enable set display-history enable set display-status enable set heading "Omeros SSL VPN" set redir-url '' set theme blue set custom-lang '' set smb-ntlmv1-auth disable set smbv1 disable set hide-sso-credential enable

next

end

 

Here is a RST frame.

 

Frame 2247: 40 bytes on wire (320 bits), 40 bytes captured (320 bits) Encapsulation type: Raw IP (7) Arrival Time: Feb 13, 2020 14:23:33.631440000 Pacific Standard Time [size="2"] [Time shift for this packet: 0.000000000 seconds][/size] Epoch Time: 1581632613.631440000 seconds [size="2"] [Time delta from previous captured frame: 1.469173000 seconds][/size] [size="2"] [Time delta from previous displayed frame: 9.668200000 seconds][/size] [size="2"] [Time since reference or first frame: 66.766862000 seconds][/size] Frame Number: 2247 Frame Length: 40 bytes (320 bits) Capture Length: 40 bytes (320 bits) [size="2"] [Frame is marked: False][/size] [size="2"] [Frame is ignored: False][/size] [size="2"] [Protocols in frame: raw:ip:tcp][/size] [size="2"] [Coloring Rule Name: TCP RST][/size] [size="2"] [Coloring Rule String: tcp.flags.reset eq 1][/size] Raw packet data Internet Protocol Version 4, Src: vpnclient.company.local (10.253.1.17), Dst: exchange.company.com (172.16.10.44) 0100 .... = Version: 4 .... 0101 = Header Length: 20 bytes (5) Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT) 0000 00.. = Differentiated Services Codepoint: Default (0) .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0) Total Length: 40 Identification: 0xe397 (58263) Flags: 0x4000, Don't fragment 0... .... .... .... = Reserved bit: Not set .1.. .... .... .... = Don't fragment: Set ..0. .... .... .... = More fragments: Not set ...0 0000 0000 0000 = Fragment offset: 0 Time to live: 128 Protocol: TCP (6) [size="2"] Header checksum: 0x54ee [validation disabled][/size] [size="2"] [Header checksum status: Unverified][/size] Source: vpnclient.company.local (10.253.1.17) Destination: exchange.company.com (172.16.10.44) Transmission Control Protocol, Src Port: 63189, Dst Port: 443, Seq: 50, Ack: 1, Len: 0 Source Port: 63189 Destination Port: 443 [size="2"] [Stream index: 28][/size] [size="2"] [TCP Segment Len: 0][/size] Sequence number: 50 (relative sequence number) [size="2"] [Next sequence number: 50 (relative sequence number)][/size] Acknowledgment number: 1 (relative ack number) 0101 .... = Header Length: 20 bytes (5) Flags: 0x014 (RST, ACK) 000. .... .... = Reserved: Not set ...0 .... .... = Nonce: Not set .... 0... .... = Congestion Window Reduced (CWR): Not set .... .0.. .... = ECN-Echo: Not set .... ..0. .... = Urgent: Not set .... ...1 .... = Acknowledgment: Set .... .... 0... = Push: Not set .... .... .1.. = Reset: Set [size="2"] [Expert Info (Warning/Sequence): Connection reset (RST)][/size] [size="2"] [Connection reset (RST)][/size] [size="2"] [Severity level: Warning][/size] [size="2"] [Group: Sequence][/size] .... .... ..0. = Syn: Not set .... .... ...0 = Fin: Not set [size="2"] [TCP Flags: ·······A·R··][/size] Window size value: 0 [size="2"] [Calculated window size: 0][/size] [size="2"] [Window size scaling factor: -1 (unknown)][/size] [size="2"] Checksum: 0xbf9e [unverified][/size] [size="2"] [Checksum Status: Unverified][/size] Urgent pointer: 0 [size="2"] [Timestamps][/size] [size="2"] [Time since first frame in this TCP stream: 19.268674000 seconds][/size] [size="2"] [Time since previous frame in this TCP stream: 9.668200000 seconds][/size]

 

We are looking at DNS possibly as an issue as records show up for multiple VPN client hostnames with the same IP, but other than that can't figure out why this is happening.  Any ideas?  Is there a good diag debug command for VPN sessions we could use?

 

3 Solutions
Toshi_Esumi
Esteemed Contributor II

If swapping back to 6.0.5 (just change the boot partition) fixes the problem, it's likely caused by one of those SSL VPN problems with 6.0.7-6.0.9 reported in this forum. Then open a case and get it verified by TAC. Depending on which one of the problems, 6.0.9 might solve the issue or might need to wait until 6.0.10 like our case.

View solution in original post

seadave

Okay, I got a fix from TAC.  Issue is 607433 related to timeouts and sessions not found.  One way you can test is to do a diag.

 

diag debug en (enable debugging)

diag debug flow filter add X.X.X.X (the IP address of the client experiencing the issue)

diag debug flow trace start 999  (999 is number of flows to capture, you may need more)

 

search for "Session mismatch" or "Session missing"

 

When done reset debug:

 

diag debug reset

diag debug disable

 

The fix is a special build FGT_XXXX-v6-build8661-FORTINET.out (your model number XXXX)  TAC can provide if you request.

 

I was on 6.0.9 so not sure if you need to be on that first.  We installed it and while I still saw occasional resets and retransmissions via WireShark on the client side, Outlook now works perfectly.  No Sychronization errors, etc.

 

Tech said that 6.0.10 and 6.2.4 should be released with the fix.

 

Thanks Fortinet support!!!

 

 

View solution in original post

rzimmermann

dfollis wrote:

grax88 wrote:

Firmware downgrade to 6.0.6 doesn't resolve the issue, i have several messages (50) in minute with dropped packet and unknown-0 destination. For tomorrow I should have remote assistance with support.

 

This is my first bigger solution (2x 100F in HA) and I am really disappointed with overall service and fortigate support.

Having used Fortigates since 2004, I would suggest you not give up based on this one situation.  There have been periods where there appears to be a wave of bad releases.  5.0 comes to mind.  Contact the TAC and see if you can obtain the 8661 build to see if that helps.  That was my mistake, I didn't call first and it caused me to waste several hours troubleshooting the issue. 

 

Always wait a few weeks when a new build is released and monitor these forums for others reporting bugs before you upgrade.  The usual advice regarding making good backups and keeping a copy of the prior firmware on hand just in case also applies.

 

The F series sound like they are supper powerful but perhaps the code for them isn't up to snuff yet.  I would agree that the one downside of Fortinet is they tend to favor getting some new products on the market before they are as stable as one would expect them to be.

 

I think in the long run you'll find they provide good value and features for the money.

Hi we have the same problem. FortiOS 6.0.8 on FGT600E-Cluster and Outlook clients have connection problems over SSL-VPN. I have tested a lot. If I use deep-inspection and a nat-pool against the exchange server it works.

 

I set this in me policy:

set tcp-mss-sender 1320

set timeout-send-rst enable

set delay-tcp-npu-session enable

 

I set this in the deep inspection profile

set rpc-over-https enable

 

I don't know why and I don't know if this works fine, but now it works for me.

 

 

   

View solution in original post

14 REPLIES 14
Toshi_Esumi
Esteemed Contributor II

If swapping back to 6.0.5 (just change the boot partition) fixes the problem, it's likely caused by one of those SSL VPN problems with 6.0.7-6.0.9 reported in this forum. Then open a case and get it verified by TAC. Depending on which one of the problems, 6.0.9 might solve the issue or might need to wait until 6.0.10 like our case.

seadave

That is what I feared, but thanks.  What problems are you seeing?

Toshi_Esumi
Esteemed Contributor II

RDP sessions through SSL VPNs drop unexpectedly with 6.0.8. But not all cases. Only when some unknown conditions are met. We were told the fix will be in 6.0.10 by TAC.

seadave

Thanks.  I've updated the code to 6.0.9 but the problem persists.  I'm begining to think this might be a DNS issue.  When a remote system with FortiClient connect its local IP AND the VPN issue IP are registered in DNS.  The other problem we have is that as clients connect/disconnect their record will remain in DNS and a new client with the re-issued IP will be added.  So we'll see things like:

 

LTP01    10.253.1.10

LTP02    10.253.1.10

LTP03    10.253.1.10

 

I've spent an insane amount of time tweaking scavenging and dynamic update settings in our 2016 AD DNS servers.  We only have this problem with Fortigate related records, I'm assuming because some sort of ack isn't being sent to indicate the record is stale.  Still digging.

seadave

Running out of options.  Have attempted to play with DTLS settings on Gate and Client (enabled on both) to no avail.  Using FC 6.0.90277.

 

Confirmed DNS caching is working as expected.

Disabled ASIC and NPU offload for SSLVPN policy with no impact.

 

Wireshark continues to show duplicate ACKs, RSTs, Out of Order, Spurious Retransmissions, when SSLVPN is up.

 

Will update our ticket.

seadave

Okay, I got a fix from TAC.  Issue is 607433 related to timeouts and sessions not found.  One way you can test is to do a diag.

 

diag debug en (enable debugging)

diag debug flow filter add X.X.X.X (the IP address of the client experiencing the issue)

diag debug flow trace start 999  (999 is number of flows to capture, you may need more)

 

search for "Session mismatch" or "Session missing"

 

When done reset debug:

 

diag debug reset

diag debug disable

 

The fix is a special build FGT_XXXX-v6-build8661-FORTINET.out (your model number XXXX)  TAC can provide if you request.

 

I was on 6.0.9 so not sure if you need to be on that first.  We installed it and while I still saw occasional resets and retransmissions via WireShark on the client side, Outlook now works perfectly.  No Sychronization errors, etc.

 

Tech said that 6.0.10 and 6.2.4 should be released with the fix.

 

Thanks Fortinet support!!!

 

 

ede_pfau
Esteemed Contributor III

Excellent work and support for all of us! Thanks a lot for your efforts, this might save a lot of headaches.


Ede

"Kernel panic: Aiee, killing interrupt handler!"
seadave

ede_pfau wrote:

Excellent work and support for all of us! Thanks a lot for your efforts, this might save a lot of headaches.

Thanks, your knowledge has helped me many times over the years. Happy to contribute.  This is one of the better vendor forums out there in my opinion due to the time and effort put in by so many end users.

grax88
New Contributor

Firmware downgrade to 6.0.6 doesn't resolve the issue, i have several messages (50) in minute with dropped packet and unknown-0 destination. For tommorow i should have remote assistance with support.

 

This is my first bigger solution (2x 100F in HA) and I am really disapointed with overall service and fortigate support.