(TLS v1.2) intermittent issue with SSL Inspection enabled

[x_member]

I'm looking for a little clarity on this after we've come across an intermittent issue on 5.2.7 with SSL Inspection enabled.

We serve a number of SSL websites to external customers from a single web server. Up until last week the server was running Windows 2008 SP2 Standard and customers had no issues accessing the site from any of the main browsers (IE, Edge, Chrome, Firefox).

Unfortunately we had an issue with the server and were forced to quickly implement a Windows 2012 R2 web server to serve the same sites. This was locked down (using IISCrypto) to offer appropriate encryption and cipher combinations, including TLS 1.2 (which was not supported on the older machine). No changes were made on the Fortigate configuration.

Since implementation we have had intermittent connectivity issues reported by customers which we have occasionally been able to replicate as they are not happening consistently. These occur across all browsers - in Firefox reporting an SSL_ERROR_BAD_MAC_ALERT when attempting to load any of the sites. NB: with ssl inspection off Firefox reports connecting using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA 256 TLS1.2.

Switching off SSL inspection for all inbound traffic to the web sites has eliminated the issue for now, however I need to understand how to diagnose and resolve the issue. My searching located an article on the Fortinet knowledge base (http://kb.fortinet.com/kb/documentLink.do?externalID=FD37726) that implies that TLS v1.2 is supported - unless I'm reading this wrong of course. Note that I'm not able to enable inspection and monitor in live as the issue seems intermittent and took (afaik) approx 18 hours to first manifest.

Any thoughts / suggestions of how to direct my investigation gratefully received.

[AndreaSoliva · Accepted solution]

Hi

Your message is actually a little bit confiusing this means: if you protect your WebServer/s internally over the FortiGate you implement actually a Reverse Proxy which is done on the FortiGate with a Virtual Server configuration. Actually the Virtual Server configuration is using in the background a normal vip object. Now what is important to know is that TLS 1.2 is not supported for SSL Offloading using Virtual Server and vip object. This means until now.....and now comes the good news!!!!

!!!!!!!!!!!!!!!!   Up to FortiOS 5.2.8 TLS 1.2 is supported on Devices which SSL Offloading can be done !!!!!!!!!!!!!!!!!

If your device is able to do SSL Offloading you see in the Software Matrix. In short words 80x, 100x and above but to be sure have a look at Software Matrix. To confgure a Virtual Server keep the picture which I attached in mind that you are fully aware how it works specially regarding the Certificate.

This means to configure a vip for the Virtual Server which is actually a Reverse Proxy (for TLS 1.2 min Version 5.2.8 must be used) do:

       # config firewall vip        # edit [Name of Virtual Server example "ActiveSync-OWA-Publishing"]        # set comment [Use a description example "Reverse Proxy ActiveSync/OWA"]        # set type server-load-balance        # unset src-filter        # unset extip        # set extip [IPv4 Public Addresse for example ActiveSync/OWA or Exchange example "193.193.135.66"]        # set extintf [Name of Interfaces for Public IPv4 Addresse for ActiveSync/OWA or Exchange zB "wan1"]        # set server-type https        # unset srcintf-filter        # unset monitor        # set persistence ssl-session-id        # unset extport        # set extport 443        # set ssl-mode full

       # set ssl-mode full        # set ssl-min-version [ssl-3.0 | tls-1.0 | tls-1.1 | tls-1.2]        # set ssl-max-version [ssl-3.0 | tls-1.0 | tls-1.1 | tls-1.2]        # set ssl-server-min-version [ssl-3.0 | tls-1.0 | tls-1.1 | tls-1.2 | client]        # set ssl-server-max-version [ssl-3.0 | tls-1.0 | tls-1.1 | tls-1.2 | client]        # set ssl-certificate [Name of Private Certificate for ActiveSync/OWA or des Exchange Servers]        # end

After that configure monitoring for the Servers within internal network or which are used over Virtual Server:

       # config firewall vip        # edit [Name of Virtual Server example "ActiveSync-OWA-Publishing"]        # config realservers        # edit [Use a integer like "0"]        # set ip [Internal IPv4 Adresse for ActiveSync/OWA or Exchange Servers 198.18.0.92        # set port 443        # unset healthcheck        # end        # end

After that configure a "ssl-ssh-profile" wich is configured as "Protecting SSL Server" and use this as the VIP within a Firewall Policy.

This is actually in short words a Reverse Proxy with SSL Offloading.

hope this helps and verifies the TLS 1.2 behaviour.

have fun

Andrea

[Willem_Bargeman · Accepted solution]

What type of inspection mode are you using on the UTM profiles? Flow or proxy based?

If the inspection mode is flow based this could be the reason for the errors. There is a know issue in the IPS engine and flow based inspection if full SSL inspection is used. A new IPS engine version will fix the issue (not included in the 5.2.8 release).

Workaround is change the inspection mode to proxy based or update the IPS engine to version 3.0284

[x_member]

Willem Bargeman wrote:

What type of inspection mode are you using on the UTM profiles? Flow or proxy based?

If the inspection mode is flow based this could be the reason for the errors. There is a know issue in the IPS engine and flow based inspection if full SSL inspection is used. A new IPS engine version will fix the issue (not included in the 5.2.8 release).

Workaround is change the inspection mode to proxy based or update the IPS engine to version 3.0284

That could well be it Willem - we use flow based and I'm currently testing with 3.282 supplied supplied by support.

You wouldn't happen to have an issue id would you? It took me over a day to get the 3.282 patch even with an id...

[Willem_Bargeman]

We experience the same issue with web browsing traffic. Simple workaround is to change to proxy mode.

I received the path last Tuesday and so far no issues.

Do you have received IPS engine version 3.284? The issue is fixed in that version. I'm not sure if the issue is fixed in version 3.282. 

[x_member]

Willem Bargeman wrote:

We experience the same issue with web browsing traffic. Simple workaround is to change to proxy mode.

I received the path last Tuesday and so far no issues.

Do you have received IPS engine version 3.284? The issue is fixed in that version. I'm not sure if the issue is fixed in version 3.282. 

I've just put in a request for it.  

[virtualj]

Hello,

I have exactly the same problem. The firewall is a 3950B with FortiOS 5.2.10. The configuration is reverse proxy with load balance, full ssl offload, tls version 1.2 and ssl inspection in the policy.

Some connections return SSL_ERROR_BAD_MAC_READ with different browsers. I see also many logs for "SSL Alert received" "SSL decryption failure" "SSL Alert sent".

This is my ldb conf:

config firewall vip
edit "am.xxxxxxx.eu"
set comment "Portalt"
set type server-load-balance
set extip x.x.x.x
set extintf "Public-2"
set server-type https
set monitor "https" "ping"
set persistence ssl-session-id
set extport 443
config realservers
edit 1
set ip 192.168.x.x
set port 443
next
edit 2
set ip 192.168.x.x
set port 443
next
end
set ssl-mode full
set ssl-certificate "cert_sites_star"
set ssl-algorithm custom
config ssl-cipher-suites
edit 1
set cipher TLS-ECDHE-ECDSA-WITH-AES-256-GCM-SHA384
next
edit 2
set cipher TLS-ECDHE-ECDSA-WITH-AES-256-CBC-SHA384
next
edit 3
set cipher TLS-ECDHE-RSA-WITH-AES-256-CBC-SHA
next
edit 4
set cipher TLS-ECDHE-ECDSA-WITH-AES-256-CBC-SHA384
next
edit 5
set cipher TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256
next
edit 6
set cipher TLS-ECDHE-ECDSA-WITH-AES-128-CBC-SHA256
next
edit 7
set cipher TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256
next
edit 8
set cipher TLS-ECDHE-ECDSA-WITH-AES-128-CBC-SHA256
next
edit 9
set cipher TLS-ECDHE-ECDSA-WITH-AES-128-CBC-SHA
next
edit 10
set cipher TLS-ECDHE-RSA-WITH-AES-128-CBC-SHA
next
edit 11
set cipher TLS-RSA-WITH-AES-256-CBC-SHA256
next
edit 12
set cipher TLS-RSA-WITH-AES-256-CBC-SHA
next
edit 13
set cipher TLS-RSA-WITH-AES-128-CBC-SHA256
next
edit 14
set cipher TLS-RSA-WITH-AES-128-CBC-SHA
next
edit 15
set cipher TLS-RSA-WITH-3DES-EDE-CBC-SHA
next
end
set ssl-min-version tls-1.0
set ssl-max-version tls-1.2
next
end
And this is ssl-ssh-inspection profile
config firewall ssl-ssh-profile
edit "Mysoc"
config https
set ports 443
end
config ftps
set ports 990
end
config imaps
set ports 993
end
config pop3s
set ports 995
end
config smtps
set ports 465
end
config ssh
set ports 22
end
set server-cert-mode replace
set server-cert "cert_sites_star"
next

config firewall policy
edit 713
set srcintf "Public-2"
set dstintf "DMZ_SERVICES"
set srcaddr "all"
set dstaddr "am.xxxxxxx.eu"
set action accept
set schedule "always"
set service "HTTPS" "PING"
set utm-status enable
set logtraffic all
set profile-protocol-options "default"
set ssl-ssh-profile "Mysoc"
set ips-sensor "am.xxxxxxx.eu"
next
end

[x_member]

virtualj wrote:

Hello,

I have exactly the same problem. The firewall is a 3950B with FortiOS 5.2.10. The configuration is reverse proxy with load balance, full ssl offload, tls version 1.2 and ssl inspection in the policy.

Some connections return SSL_ERROR_BAD_MAC_READ with different browsers. I see also many logs for "SSL Alert received" "SSL decryption failure" "SSL Alert sent".

<snip>

Interesting that you get log alerts - we have none.

We've had to remain running with SSL inspection disabled as it was making our services unreliable for clients, while pushing for a fix via TAC management.

We've been informed that 5.2.11 with IPSEngine v3.310 will fix our issue - we have the patch scheduled for install in early June pending user feedback here on the stability of the firmware.