Outlook Clients Connection Errors 6.0.7 SSL-VPN Full Tunnel

[seadave]

Hello-

Yesterday we updated our HA 501E pair from 6.0.5 to 6.0.7.  All seemed to be fine, but today, users connected to our SSL VPN tunnel are having Outlook 2016 Connectivity issues.  We have Outlook on-prem and only clients coming in over the SSL VPN full tunnel are having the issue.

Outlook sync log keeps generating errors similar to:

14:58:32 Synchronizer Version 16.0.11328 14:58:32 Synchronizing Mailbox 'Joe User' 14:58:32 Synchronizing Hierarchy 14:58:33 Synchronizing server changes in folder 'Inbox' 14:58:33 Downloading from server 'd10db7e0-492a-4521-8c0a-fe49225ad443@company.com' 14:58:34 Downloading from server 'd10db7e0-492a-4521-8c0a-fe49225ad443@company.com' 14:58:34 Error synchronizing folder [size="2"]14:58:34 [80040115-514-80040115-0][/size] 14:58:34 Network problems are preventing connection to Microsoft Exchange. 14:58:34 Microsoft Exchange Information Store 14:58:34 For more information on this failure, click the URL below: 14:58:34 https://www.microsoft.com...0040115-514-80040115-0 14:58:34 Done 14:58:34 Microsoft Exchange offline address book 14:58:34 Download successful

Of course the support link goes to a generic MS help page and is unrelated.

If we put any of our clients on a mobile hotspot and VPN in, this happens.  If we take them off and reconnect to wifi, no more errors.  So it seems like something is happening to the traffic as it passes through the tunnel.

We are using a full tunnel, without split DNS.  Here are the settings:

config vpn ssl settings set reqclientcert disable set tlsv1-0 disable set tlsv1-1 disable set tlsv1-2 enable unset banned-cipher set ssl-insert-empty-fragment enable set https-redirect enable set x-content-type-options enable set ssl-client-renegotiation enable set force-two-factor-auth disable [size="2"] set servercert "XXXXXXXX"  (Obscured for post)[/size] set algorithm high set idle-timeout 7200 set auth-timeout 28800 set login-attempt-limit 2 set login-block-time 60 set login-timeout 30 set dtls-hello-timeout 10 set tunnel-ip-pools "SSLVPN_TUNNEL_ADDR1" set tunnel-ipv6-pools "SSLVPN_TUNNEL_IPv6_ADDR1" set dns-suffix "company.local" set dns-server1 XXXXXXXXXXX (Obscured for post) [size="2"] set dns-server2 XXXXXXXXXXX (Obscured for post)[/size] set wins-server1 0.0.0.0 set wins-server2 0.0.0.0 set ipv6-dns-server1 :: set ipv6-dns-server2 :: set ipv6-wins-server1 :: set ipv6-wins-server2 :: set url-obscuration enable set http-compression disable set http-only-cookie enable set port 443 set port-precedence enable set auto-tunnel-static-route enable set header-x-forwarded-for add set source-interface "WAN" set source-address "all" set source-address-negate disable set source-address6 "all" set source-address6-negate disable set default-portal "SSL_VPN_TS" config authentication-rule edit 8 set groups "SSL_VPN_FULL" set portal "SSL_VPN_FULL" set realm '' set client-cert disable set cipher high set auth any next

set dtls-tunnel enable set check-referer enable set http-request-header-timeout 20 set http-request-body-timeout 30 set unsafe-legacy-renegotiation disable

end

We attempted to sniff the traffic and the PCAPS do show RSTs occurring.

We have used the Default Web Proxy for ever with these connections and tried to use a custom one with RPC over HTTP enabled but it made no change to the behavior.

config vpn ssl web portal edit "SSL_VPN_FULL" set tunnel-mode enable [size="2"] set ipv6-tunnel-mode disable [style="background-color: #ffff00;"]<----- This was enabled and we set to disabled, but no luck[/style][/size] set web-mode enable set host-check none set limit-user-logins enable set mac-addr-check disable set os-check disable set forticlient-download disable set ip-mode range set auto-connect disable set keep-alive disable set save-password disable set ip-pools "SSLVPN_TUNNEL_ADDR1" set exclusive-routing disable set service-restriction disable set split-tunneling disable set dns-server1 0.0.0.0 set dns-server2 0.0.0.0 set dns-suffix '' set wins-server1 0.0.0.0 set wins-server2 0.0.0.0 set display-bookmark enable set user-bookmark enable set allow-user-access web ftp smb telnet ssh vnc rdp ping citrix portforward set user-group-bookmark enable

set display-connection-tools enable set display-history enable set display-status enable set heading "Omeros SSL VPN" set redir-url '' set theme blue set custom-lang '' set smb-ntlmv1-auth disable set smbv1 disable set hide-sso-credential enable

next

end

Here is a RST frame.

Frame 2247: 40 bytes on wire (320 bits), 40 bytes captured (320 bits) Encapsulation type: Raw IP (7) Arrival Time: Feb 13, 2020 14:23:33.631440000 Pacific Standard Time [size="2"] [Time shift for this packet: 0.000000000 seconds][/size] Epoch Time: 1581632613.631440000 seconds [size="2"] [Time delta from previous captured frame: 1.469173000 seconds][/size] [size="2"] [Time delta from previous displayed frame: 9.668200000 seconds][/size] [size="2"] [Time since reference or first frame: 66.766862000 seconds][/size] Frame Number: 2247 Frame Length: 40 bytes (320 bits) Capture Length: 40 bytes (320 bits) [size="2"] [Frame is marked: False][/size] [size="2"] [Frame is ignored: False][/size] [size="2"] [Protocols in frame: raw:ip:tcp][/size] [size="2"] [Coloring Rule Name: TCP RST][/size] [size="2"] [Coloring Rule String: tcp.flags.reset eq 1][/size] Raw packet data Internet Protocol Version 4, Src: vpnclient.company.local (10.253.1.17), Dst: exchange.company.com (172.16.10.44) 0100 .... = Version: 4 .... 0101 = Header Length: 20 bytes (5) Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT) 0000 00.. = Differentiated Services Codepoint: Default (0) .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0) Total Length: 40 Identification: 0xe397 (58263) Flags: 0x4000, Don't fragment 0... .... .... .... = Reserved bit: Not set .1.. .... .... .... = Don't fragment: Set ..0. .... .... .... = More fragments: Not set ...0 0000 0000 0000 = Fragment offset: 0 Time to live: 128 Protocol: TCP (6) [size="2"] Header checksum: 0x54ee [validation disabled][/size] [size="2"] [Header checksum status: Unverified][/size] Source: vpnclient.company.local (10.253.1.17) Destination: exchange.company.com (172.16.10.44) Transmission Control Protocol, Src Port: 63189, Dst Port: 443, Seq: 50, Ack: 1, Len: 0 Source Port: 63189 Destination Port: 443 [size="2"] [Stream index: 28][/size] [size="2"] [TCP Segment Len: 0][/size] Sequence number: 50 (relative sequence number) [size="2"] [Next sequence number: 50 (relative sequence number)][/size] Acknowledgment number: 1 (relative ack number) 0101 .... = Header Length: 20 bytes (5) Flags: 0x014 (RST, ACK) 000. .... .... = Reserved: Not set ...0 .... .... = Nonce: Not set .... 0... .... = Congestion Window Reduced (CWR): Not set .... .0.. .... = ECN-Echo: Not set .... ..0. .... = Urgent: Not set .... ...1 .... = Acknowledgment: Set .... .... 0... = Push: Not set .... .... .1.. = Reset: Set [size="2"] [Expert Info (Warning/Sequence): Connection reset (RST)][/size] [size="2"] [Connection reset (RST)][/size] [size="2"] [Severity level: Warning][/size] [size="2"] [Group: Sequence][/size] .... .... ..0. = Syn: Not set .... .... ...0 = Fin: Not set [size="2"] [TCP Flags: ·······A·R··][/size] Window size value: 0 [size="2"] [Calculated window size: 0][/size] [size="2"] [Window size scaling factor: -1 (unknown)][/size] [size="2"] Checksum: 0xbf9e [unverified][/size] [size="2"] [Checksum Status: Unverified][/size] Urgent pointer: 0 [size="2"] [Timestamps][/size] [size="2"] [Time since first frame in this TCP stream: 19.268674000 seconds][/size] [size="2"] [Time since previous frame in this TCP stream: 9.668200000 seconds][/size]

We are looking at DNS possibly as an issue as records show up for multiple VPN client hostnames with the same IP, but other than that can't figure out why this is happening.  Any ideas?  Is there a good diag debug command for VPN sessions we could use?

[Toshi_Esumi · Accepted solution]

If swapping back to 6.0.5 (just change the boot partition) fixes the problem, it's likely caused by one of those SSL VPN problems with 6.0.7-6.0.9 reported in this forum. Then open a case and get it verified by TAC. Depending on which one of the problems, 6.0.9 might solve the issue or might need to wait until 6.0.10 like our case.

[seadave · Accepted solution]

Okay, I got a fix from TAC.  Issue is 607433 related to timeouts and sessions not found.  One way you can test is to do a diag.

diag debug en (enable debugging)

diag debug flow filter add X.X.X.X (the IP address of the client experiencing the issue)

diag debug flow trace start 999  (999 is number of flows to capture, you may need more)

search for "Session mismatch" or "Session missing"

When done reset debug:

diag debug reset

diag debug disable

The fix is a special build FGT_XXXX-v6-build8661-FORTINET.out (your model number XXXX)  TAC can provide if you request.

I was on 6.0.9 so not sure if you need to be on that first.  We installed it and while I still saw occasional resets and retransmissions via WireShark on the client side, Outlook now works perfectly.  No Sychronization errors, etc.

Tech said that 6.0.10 and 6.2.4 should be released with the fix.

Thanks Fortinet support!!!

[Toshi_Esumi · Accepted solution]

If swapping back to 6.0.5 (just change the boot partition) fixes the problem, it's likely caused by one of those SSL VPN problems with 6.0.7-6.0.9 reported in this forum. Then open a case and get it verified by TAC. Depending on which one of the problems, 6.0.9 might solve the issue or might need to wait until 6.0.10 like our case.

[seadave]

That is what I feared, but thanks.  What problems are you seeing?

[Toshi_Esumi]

RDP sessions through SSL VPNs drop unexpectedly with 6.0.8. But not all cases. Only when some unknown conditions are met. We were told the fix will be in 6.0.10 by TAC.

[seadave]

Thanks.  I've updated the code to 6.0.9 but the problem persists.  I'm begining to think this might be a DNS issue.  When a remote system with FortiClient connect its local IP AND the VPN issue IP are registered in DNS.  The other problem we have is that as clients connect/disconnect their record will remain in DNS and a new client with the re-issued IP will be added.  So we'll see things like:

LTP01    10.253.1.10

LTP02    10.253.1.10

LTP03    10.253.1.10

I've spent an insane amount of time tweaking scavenging and dynamic update settings in our 2016 AD DNS servers.  We only have this problem with Fortigate related records, I'm assuming because some sort of ack isn't being sent to indicate the record is stale.  Still digging.

[seadave]

Running out of options.  Have attempted to play with DTLS settings on Gate and Client (enabled on both) to no avail.  Using FC 6.0.90277.

Confirmed DNS caching is working as expected.

Disabled ASIC and NPU offload for SSLVPN policy with no impact.

Wireshark continues to show duplicate ACKs, RSTs, Out of Order, Spurious Retransmissions, when SSLVPN is up.

Will update our ticket.

[seadave · Accepted solution]

Okay, I got a fix from TAC.  Issue is 607433 related to timeouts and sessions not found.  One way you can test is to do a diag.

diag debug en (enable debugging)

diag debug flow filter add X.X.X.X (the IP address of the client experiencing the issue)

diag debug flow trace start 999  (999 is number of flows to capture, you may need more)

search for "Session mismatch" or "Session missing"

When done reset debug:

diag debug reset

diag debug disable

The fix is a special build FGT_XXXX-v6-build8661-FORTINET.out (your model number XXXX)  TAC can provide if you request.

I was on 6.0.9 so not sure if you need to be on that first.  We installed it and while I still saw occasional resets and retransmissions via WireShark on the client side, Outlook now works perfectly.  No Sychronization errors, etc.

Tech said that 6.0.10 and 6.2.4 should be released with the fix.

Thanks Fortinet support!!!

[ede_pfau]

Excellent work and support for all of us! Thanks a lot for your efforts, this might save a lot of headaches.

[seadave]

ede_pfau wrote:

Excellent work and support for all of us! Thanks a lot for your efforts, this might save a lot of headaches.

Thanks, your knowledge has helped me many times over the years. Happy to contribute.  This is one of the better vendor forums out there in my opinion due to the time and effort put in by so many end users.

[grax88]

Firmware downgrade to 6.0.6 doesn't resolve the issue, i have several messages (50) in minute with dropped packet and unknown-0 destination. For tommorow i should have remote assistance with support.

This is my first bigger solution (2x 100F in HA) and I am really disapointed with overall service and fortigate support.