Outlook Clients Connection Errors 6.0.7 SSL-VPN Full Tunnel

[seadave]

Hello-

Yesterday we updated our HA 501E pair from 6.0.5 to 6.0.7.  All seemed to be fine, but today, users connected to our SSL VPN tunnel are having Outlook 2016 Connectivity issues.  We have Outlook on-prem and only clients coming in over the SSL VPN full tunnel are having the issue.

Outlook sync log keeps generating errors similar to:

14:58:32 Synchronizer Version 16.0.11328 14:58:32 Synchronizing Mailbox 'Joe User' 14:58:32 Synchronizing Hierarchy 14:58:33 Synchronizing server changes in folder 'Inbox' 14:58:33 Downloading from server 'd10db7e0-492a-4521-8c0a-fe49225ad443@company.com' 14:58:34 Downloading from server 'd10db7e0-492a-4521-8c0a-fe49225ad443@company.com' 14:58:34 Error synchronizing folder [size="2"]14:58:34 [80040115-514-80040115-0][/size] 14:58:34 Network problems are preventing connection to Microsoft Exchange. 14:58:34 Microsoft Exchange Information Store 14:58:34 For more information on this failure, click the URL below: 14:58:34 https://www.microsoft.com...0040115-514-80040115-0 14:58:34 Done 14:58:34 Microsoft Exchange offline address book 14:58:34 Download successful

Of course the support link goes to a generic MS help page and is unrelated.

If we put any of our clients on a mobile hotspot and VPN in, this happens.  If we take them off and reconnect to wifi, no more errors.  So it seems like something is happening to the traffic as it passes through the tunnel.

We are using a full tunnel, without split DNS.  Here are the settings:

config vpn ssl settings set reqclientcert disable set tlsv1-0 disable set tlsv1-1 disable set tlsv1-2 enable unset banned-cipher set ssl-insert-empty-fragment enable set https-redirect enable set x-content-type-options enable set ssl-client-renegotiation enable set force-two-factor-auth disable [size="2"] set servercert "XXXXXXXX"  (Obscured for post)[/size] set algorithm high set idle-timeout 7200 set auth-timeout 28800 set login-attempt-limit 2 set login-block-time 60 set login-timeout 30 set dtls-hello-timeout 10 set tunnel-ip-pools "SSLVPN_TUNNEL_ADDR1" set tunnel-ipv6-pools "SSLVPN_TUNNEL_IPv6_ADDR1" set dns-suffix "company.local" set dns-server1 XXXXXXXXXXX (Obscured for post) [size="2"] set dns-server2 XXXXXXXXXXX (Obscured for post)[/size] set wins-server1 0.0.0.0 set wins-server2 0.0.0.0 set ipv6-dns-server1 :: set ipv6-dns-server2 :: set ipv6-wins-server1 :: set ipv6-wins-server2 :: set url-obscuration enable set http-compression disable set http-only-cookie enable set port 443 set port-precedence enable set auto-tunnel-static-route enable set header-x-forwarded-for add set source-interface "WAN" set source-address "all" set source-address-negate disable set source-address6 "all" set source-address6-negate disable set default-portal "SSL_VPN_TS" config authentication-rule edit 8 set groups "SSL_VPN_FULL" set portal "SSL_VPN_FULL" set realm '' set client-cert disable set cipher high set auth any next

set dtls-tunnel enable set check-referer enable set http-request-header-timeout 20 set http-request-body-timeout 30 set unsafe-legacy-renegotiation disable

end

We attempted to sniff the traffic and the PCAPS do show RSTs occurring.

We have used the Default Web Proxy for ever with these connections and tried to use a custom one with RPC over HTTP enabled but it made no change to the behavior.

config vpn ssl web portal edit "SSL_VPN_FULL" set tunnel-mode enable [size="2"] set ipv6-tunnel-mode disable [style="background-color: #ffff00;"]<----- This was enabled and we set to disabled, but no luck[/style][/size] set web-mode enable set host-check none set limit-user-logins enable set mac-addr-check disable set os-check disable set forticlient-download disable set ip-mode range set auto-connect disable set keep-alive disable set save-password disable set ip-pools "SSLVPN_TUNNEL_ADDR1" set exclusive-routing disable set service-restriction disable set split-tunneling disable set dns-server1 0.0.0.0 set dns-server2 0.0.0.0 set dns-suffix '' set wins-server1 0.0.0.0 set wins-server2 0.0.0.0 set display-bookmark enable set user-bookmark enable set allow-user-access web ftp smb telnet ssh vnc rdp ping citrix portforward set user-group-bookmark enable

set display-connection-tools enable set display-history enable set display-status enable set heading "Omeros SSL VPN" set redir-url '' set theme blue set custom-lang '' set smb-ntlmv1-auth disable set smbv1 disable set hide-sso-credential enable

next

end

Here is a RST frame.

Frame 2247: 40 bytes on wire (320 bits), 40 bytes captured (320 bits) Encapsulation type: Raw IP (7) Arrival Time: Feb 13, 2020 14:23:33.631440000 Pacific Standard Time [size="2"] [Time shift for this packet: 0.000000000 seconds][/size] Epoch Time: 1581632613.631440000 seconds [size="2"] [Time delta from previous captured frame: 1.469173000 seconds][/size] [size="2"] [Time delta from previous displayed frame: 9.668200000 seconds][/size] [size="2"] [Time since reference or first frame: 66.766862000 seconds][/size] Frame Number: 2247 Frame Length: 40 bytes (320 bits) Capture Length: 40 bytes (320 bits) [size="2"] [Frame is marked: False][/size] [size="2"] [Frame is ignored: False][/size] [size="2"] [Protocols in frame: raw:ip:tcp][/size] [size="2"] [Coloring Rule Name: TCP RST][/size] [size="2"] [Coloring Rule String: tcp.flags.reset eq 1][/size] Raw packet data Internet Protocol Version 4, Src: vpnclient.company.local (10.253.1.17), Dst: exchange.company.com (172.16.10.44) 0100 .... = Version: 4 .... 0101 = Header Length: 20 bytes (5) Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT) 0000 00.. = Differentiated Services Codepoint: Default (0) .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0) Total Length: 40 Identification: 0xe397 (58263) Flags: 0x4000, Don't fragment 0... .... .... .... = Reserved bit: Not set .1.. .... .... .... = Don't fragment: Set ..0. .... .... .... = More fragments: Not set ...0 0000 0000 0000 = Fragment offset: 0 Time to live: 128 Protocol: TCP (6) [size="2"] Header checksum: 0x54ee [validation disabled][/size] [size="2"] [Header checksum status: Unverified][/size] Source: vpnclient.company.local (10.253.1.17) Destination: exchange.company.com (172.16.10.44) Transmission Control Protocol, Src Port: 63189, Dst Port: 443, Seq: 50, Ack: 1, Len: 0 Source Port: 63189 Destination Port: 443 [size="2"] [Stream index: 28][/size] [size="2"] [TCP Segment Len: 0][/size] Sequence number: 50 (relative sequence number) [size="2"] [Next sequence number: 50 (relative sequence number)][/size] Acknowledgment number: 1 (relative ack number) 0101 .... = Header Length: 20 bytes (5) Flags: 0x014 (RST, ACK) 000. .... .... = Reserved: Not set ...0 .... .... = Nonce: Not set .... 0... .... = Congestion Window Reduced (CWR): Not set .... .0.. .... = ECN-Echo: Not set .... ..0. .... = Urgent: Not set .... ...1 .... = Acknowledgment: Set .... .... 0... = Push: Not set .... .... .1.. = Reset: Set [size="2"] [Expert Info (Warning/Sequence): Connection reset (RST)][/size] [size="2"] [Connection reset (RST)][/size] [size="2"] [Severity level: Warning][/size] [size="2"] [Group: Sequence][/size] .... .... ..0. = Syn: Not set .... .... ...0 = Fin: Not set [size="2"] [TCP Flags: ·······A·R··][/size] Window size value: 0 [size="2"] [Calculated window size: 0][/size] [size="2"] [Window size scaling factor: -1 (unknown)][/size] [size="2"] Checksum: 0xbf9e [unverified][/size] [size="2"] [Checksum Status: Unverified][/size] Urgent pointer: 0 [size="2"] [Timestamps][/size] [size="2"] [Time since first frame in this TCP stream: 19.268674000 seconds][/size] [size="2"] [Time since previous frame in this TCP stream: 9.668200000 seconds][/size]

We are looking at DNS possibly as an issue as records show up for multiple VPN client hostnames with the same IP, but other than that can't figure out why this is happening.  Any ideas?  Is there a good diag debug command for VPN sessions we could use?

[Toshi_Esumi · Accepted solution]

If swapping back to 6.0.5 (just change the boot partition) fixes the problem, it's likely caused by one of those SSL VPN problems with 6.0.7-6.0.9 reported in this forum. Then open a case and get it verified by TAC. Depending on which one of the problems, 6.0.9 might solve the issue or might need to wait until 6.0.10 like our case.

[seadave · Accepted solution]

Okay, I got a fix from TAC.  Issue is 607433 related to timeouts and sessions not found.  One way you can test is to do a diag.

diag debug en (enable debugging)

diag debug flow filter add X.X.X.X (the IP address of the client experiencing the issue)

diag debug flow trace start 999  (999 is number of flows to capture, you may need more)

search for "Session mismatch" or "Session missing"

When done reset debug:

diag debug reset

diag debug disable

The fix is a special build FGT_XXXX-v6-build8661-FORTINET.out (your model number XXXX)  TAC can provide if you request.

I was on 6.0.9 so not sure if you need to be on that first.  We installed it and while I still saw occasional resets and retransmissions via WireShark on the client side, Outlook now works perfectly.  No Sychronization errors, etc.

Tech said that 6.0.10 and 6.2.4 should be released with the fix.

Thanks Fortinet support!!!

[seadave]

grax88 wrote:

Firmware downgrade to 6.0.6 doesn't resolve the issue, i have several messages (50) in minute with dropped packet and unknown-0 destination. For tomorrow I should have remote assistance with support.

This is my first bigger solution (2x 100F in HA) and I am really disappointed with overall service and fortigate support.

Having used Fortigates since 2004, I would suggest you not give up based on this one situation.  There have been periods where there appears to be a wave of bad releases.  5.0 comes to mind.  Contact the TAC and see if you can obtain the 8661 build to see if that helps.  That was my mistake, I didn't call first and it caused me to waste several hours troubleshooting the issue. 

Always wait a few weeks when a new build is released and monitor these forums for others reporting bugs before you upgrade.  The usual advice regarding making good backups and keeping a copy of the prior firmware on hand just in case also applies.

The F series sound like they are supper powerful but perhaps the code for them isn't up to snuff yet.  I would agree that the one downside of Fortinet is they tend to favor getting some new products on the market before they are as stable as one would expect them to be.

I think in the long run you'll find they provide good value and features for the money.

[rzimmermann · Accepted solution]

dfollis wrote:

grax88 wrote:

Firmware downgrade to 6.0.6 doesn't resolve the issue, i have several messages (50) in minute with dropped packet and unknown-0 destination. For tomorrow I should have remote assistance with support.

This is my first bigger solution (2x 100F in HA) and I am really disappointed with overall service and fortigate support.

Having used Fortigates since 2004, I would suggest you not give up based on this one situation.  There have been periods where there appears to be a wave of bad releases.  5.0 comes to mind.  Contact the TAC and see if you can obtain the 8661 build to see if that helps.  That was my mistake, I didn't call first and it caused me to waste several hours troubleshooting the issue. 

Always wait a few weeks when a new build is released and monitor these forums for others reporting bugs before you upgrade.  The usual advice regarding making good backups and keeping a copy of the prior firmware on hand just in case also applies.

The F series sound like they are supper powerful but perhaps the code for them isn't up to snuff yet.  I would agree that the one downside of Fortinet is they tend to favor getting some new products on the market before they are as stable as one would expect them to be.

I think in the long run you'll find they provide good value and features for the money.

Hi we have the same problem. FortiOS 6.0.8 on FGT600E-Cluster and Outlook clients have connection problems over SSL-VPN. I have tested a lot. If I use deep-inspection and a nat-pool against the exchange server it works.

I set this in me policy:

set tcp-mss-sender 1320

set timeout-send-rst enable

set delay-tcp-npu-session enable

I set this in the deep inspection profile

set rpc-over-https enable

I don't know why and I don't know if this works fine, but now it works for me.

   

[seadave]

Nice alternate solution.  I just became aware of the set tcp-mss-sender parameter when setting up some IPSec tunnels.  Good to know it might work here.

[grax88]

Provided by support firmware 8660 (for FG 100F), RDP retransmission fixed but still problem with HTTPS. There is lot 

dstintf="unknown-0" msg="no session matched"

messages. I was asked to downgrade do 6.0.6.

[seadave]

grax88 wrote:

Provided by support firmware 8660 (for FG 100F), RDP retransmission fixed but still problem with HTTPS. There is lot 

dstintf="unknown-0" msg="no session matched"

messages. I was asked to downgrade do 6.0.6.

Thanks for posting the proper log message display.  I didn't catch it when working with TAC.  "no session matched" NOT missing.  Sorry.

Note the build I was issued was 8661 which is newer than what they provided you with.  Also Fs are the newest chip-set.  I find it interesting that the problem spans E and F revisions.  Suggests it is more of a coding than hardware issue. 

I was watching a laptop connected via SSLVPN over a mobile hotspot with Wireshark real-time as I updated.  As soon as I applied this build and 501Es rebooted (so nice that HA doesn't drop VPN sessions!!!), the number of RST and re-transmission ACKs dropped by 80% (best guess).  It did NOT eliminate them, but Outlook no longer produced Synchronization errors every few minutes and I was able to send/receive messages without delay.

So I agree some low level TCP packet issues may remain, but 8661 was WAY better than 6.0.9GA.  In WireShark, the packets marked TLSv1.2 have no issues (I assume that is cert negotiations) whereas the TCP protocols are the ones that show re-transmissions.  Need to look into that more.

We went from 6.0.5 to 6.0.7 initially, maybe we should have tried .6 first.

[Faiza_Emam_Delhi]

The Outlook connectivity issues you are experiencing with SSL VPN Full Tunnel users could be related to a number of factors. Here are some potential causes and solutions:

1. SSL VPN configuration: The SSL VPN configuration may have changed during the upgrade to 6.0.7, causing connectivity issues. You may want to review the SSL VPN configuration and ensure that it is set up correctly.

2. Firewall rules: The FortiGate firewall rules may be blocking the traffic required for Outlook connectivity. You may need to create new firewall rules to allow the required traffic to pass through the SSL VPN.

3. DNS resolution: The SSL VPN clients may be having issues with DNS resolution, causing connectivity issues with Outlook. You may want to check the DNS settings on the SSL VPN configuration and ensure that they are set up correctly.

4. Outlook client settings: The Outlook clients may have incorrect settings or configurations, causing connectivity issues. You may want to review the Outlook client settings and ensure that they are set up correctly.

5. IP address conflicts: The SSL VPN clients may be conflicting with internal IP addresses, causing connectivity issues with Outlook. You may want to check the IP address settings on the SSL VPN configuration and ensure that they are not conflicting with internal IP addresses.

It may be helpful to review the FortiGate logs to see if any traffic is being blocked or filtered. Additionally, you may want to try connecting to the SSL VPN from a different network or device to see if the issue is specific to certain users or devices.