SSL Inspection policy failing to use new certificate

[x_member]

Fortigate 60D v5.2.11

We've had cause to re-issue the certificate that we use for deep inspection on outbound traffic (moving from SHA-1 to SHA-256). This certificate has been installed as trusted on all affected internal clients.

To try and ensure a smooth transition, I've installed the new certificate and set it for use in a cloned copy of the outbound deep inspection SSL inspection policy, leaving the original SHA-1 certificate used by the original outbound inspection policy.

I've switched one of the firewall rules for traffic to use the new SSL inspection policy with the SHA-256 certificate. 

That was 50 minutes ago, however the fortigate is still utilising the original SHA-15 certificate for inspection (as checking any inspected HTTPS website certificate chain confirms). This behaviour is replicated across multiple clients and browsers (IE11, Chrome, Firefox).

I'm assuming that the Fortigate has cached the old certificate. 

Rebooting the firewall is not an option.

How do I 'encourage' the firewall to respect this configuration change and begin using the new SHA-256 certificate to inspect outbound traffic? Is there a process I can restart without interfering with other functionality / traffic?

[emnoc]

Dumb question, is the ssl-certificate removed from the  appliance?

What does diag debug flow show ?

I just test my own firewall and nothing is being cached btw.

[x_member]

emnoc wrote:

Dumb question, is the ssl-certificate removed from the  appliance?

What does diag debug flow show ?

I just test my own firewall and nothing is being cached btw.

Yes - 100% removed as of about 10am this morning.

No diag debug flow output as I'm not at work until Monday morning .

When browsing to inspected sites, their certificate chain still shows the old (removed) certificate some 6 hours+ after it was removed from the appliance. Removing the old certificate from trusted root store on the client results in non-functional browsing.

I've cleared every cache known to man on my laptop and the issue persists, however I found one or two (seldom visited) HTTPS sites using the new certificate. This is what lead me to think caching might be at play, along with the previous existence of a CLI command specifically to clear the SSL cache.

All affected machines currently have both old and new certificates in their trusted store, so I believe nothing should break while this state continues.

Will have to see if time cures it, otherwise I may have a small window to reboot later in the week if I can get to the office early enough. 

[emnoc]

Okay your probably right and it'c cachcing on the  browser can you do  the following

1: find a macosx and  run gnutls-cli <site name> does the  old chain or new chain shows up?

2: on a browser FIREFOX  do  refresh  ( here's link

https://support.mozilla.org/en-US/kb/refresh-firefox-reset-add-ons-and-settings?redirectlocale=en-US...

3: now run a site and do you see the old or new chain?

4: if you have chrome they have an optional to refresh also

try that but the fortigate is not the culprit in this case. Report back on what you discovery using step #1 or #2 or #3

Thanks

Ken

[x_member]

emnoc wrote:

Report back on what you discovery using step #1 or #2 or #3

All browser caches were cleared during testing, however..

1. No mac os available

2. Firefox (which I have installed but only use for testing internally developed sites and services) reset completed and certificates reimported as trusted into Firefox. Firefox continues to use the SHA1 certificate that is no longer present on the FGT appliance.

However, if I pick a page never before visited, e.g. https://support.mozilla.org/en-US/questions/995117 then the certificate chain uses the new SHA256 certificate.

I've also dug out an unused spare laptop and tried accessing sites from it that I have accessed before from my usual laptop (but not the spare); these also show the SHA1 certificate in the chain. Accessing sites never before visited show the new SHA256 certificate.

I think at this point I'm going to have to find a way to schedule a reboot of the FGT over the course of the week - my suspicion is that this will solve the problem; I came across some old configuration notes from 5.2.2 / 5.2.3 that imply I had this issue previously and resolved it only with a reboot.

[emnoc]

confused

So the FF reset did not work or you didn't do if?

If you take a brand new "laptop / desktop" and connect  it thru the fortigate does it show the correct cert  in the chain?

Your problem is not the  FGT btw, think about it. If the cert is out of the SSL-inspection  and a client is  seeing the old cert, it's cached by the client.

If you  don't have have gnutls-cli than use  curl ( it's available on  windows but my exprience is limited in windoze env ;) )

read this blog , it's a BC-SGproxy but the same principle applies. We have various  clients using  the proxy and various sites and client had the trust-EnterpriseOrgCA in the chain. I even use forum.fortinet.com in my example ;)

http://socpuppet.blogspot.com/2017/11/ssl-cert-caching-for-mitm-inspections.html

 BTW: I should re-label the title  since the caching is not the MiTM device , it's the client.

[x_member]

emnoc wrote:

So the FF reset did not work or you didn't do if?

As above, I did the reset in Firefox. It made no difference to the behaviour.

emnoc wrote:

If you take a brand new "laptop / desktop" and connect  it thru the fortigate does it show the correct cert  in the chain?

I have no more machines to test with.

emnoc wrote:

Your problem is not the  FGT btw, think about it. If the cert is out of the SSL-inspection  and a client is  seeing the old cert, it's cached by the client.

I'm sure you're right, however I have cleared all accessible browser caches in my attempts to resolve this and a Firefox installation reset per instructions still exhibits the same behaviour as before. 

I am now out of time to spend on this issue for the moment, have no additional resources to test with, and no opportunity to reboot the FGT before planned maintenance early next month.

If the problem is some hidden and inaccessible caching on user machines I would hope that it would clear automatically before then. Failing that, I'll see what a reboot brings.

[emnoc]

"browser caches"

This is not a browser  cache issues. It's CA-store for website clearing the browser does not fix this in  FF.

What happens when your  hit the site the web-browser checks the  CAchain and any OSCP/CRL issues and caches this. This cause the  "servercert+CAchain issuer" to be cache and no more looks. Chrome is different, I don't use  MSIE so can't speak 100% on that browser.

When you remove the MiTM  device out of the picture ( your fortigate , or in my case  BCPROXYSG ) the  cache CAissues is falsely  display by the "web-client-browser". This is why tools like GNUtls and Curl are helpful.

Reseting the  browser SSL-CA cache or forcing the  web-browser to perform a look and not cache  could be used.

Try this, the system(s) you have problems on ( a laptop )

  1; While on the network,  that has  the SSL inspection,  goto a website that you never been to....

  2; read the browser ca-chain ( it will shows the CA certificate that you  trust and the MiTM devices uses .) Right ?

  3; now take that  LAPTOP device down  and goto off-network   Wifi/Mifi,starbuck,homedepot,walmart and go to the same  exact  website ? The CAchain and issuer will be the same value you have notice in step#2 .Right ? and with your  certificate from the  MITM device.

  4; You can also see this behavior  in FF, by going to security >  certificates > and reviewing the site certificate that are cache

  5;  in Chrome ( which exhibits the right ca-chain always  regardless if your  on or off-net  ) you go to developer-tool > security,  and click the certificate details. It will always show you the  correct chain for if your on-net with a MiTM device or not from my experiences

Again it's not the MiTM it's how your browser cache  SSL-chain/issue details and what inspect if the device is on or off-net.

Ken

[x_member]

emnoc wrote:

Try this, the system(s) you have problems on ( a laptop )

  1; While on the network,  that has  the SSL inspection,  goto a website that you never been to....

  2; read the browser ca-chain ( it will shows the CA certificate that you  trust and the MiTM devices uses .) Right ?

  3; now take that  LAPTOP device down  and goto off-network   Wifi/Mifi,starbuck,homedepot,walmart and go to the same  exact  website ? The CAchain and issuer will be the same value you have notice in step#2 .Right ? and with your  certificate from the  MITM device.

Unfortunately, using Chrome and our guest wifi (that sits outside the FGT's sphere of influence) and testing with https://www.bbc.co.uk that is not what I see when I follow the above.

BBC site is chained from our original SHA1 cert when visited inside from behind the FGT. After switching to guest wifi and reloading the BBC site I see the chain terminating against a GlobalSign cert.

Regardless I am out of time to mess with this situation. Thank you for your assistance Ken.

I will update this thread again when I get more time to look at this again. For the moment it is an irritant but no a show-stopper (as we have both old SHA1 and new SHA256 loaded on all affected machines). FGT reboot is scheduled for early next month in our next maintenance window.

[x_member]

To update with a resolution.

The issue persisted until we performed a reboot on the Fortigate appliance during our maintenance window yesterday.

This immediately resolved the issue across all machines for all external sites (including those previously visited).