Deep Packet Inspection pinning ?

[fran1942]

Hello, I have just implemented Deep Packet SSL Inspection on our Fortigate firewall.

I am finding instances of SSL certificate pinning (HPKP) where I need to make exceptions to the DPI list e.g. *.google.com etc.

This fixes the problem.

What I am finding strange is how some of the sites I need to make exceptions for do not 'seem' to be using HPKP pinning (or HSTS.

For example, I look within Chrome browser 'chrome://net-internals/#capture' I do not see any entries for those particular sites using pinning or HSTS (HTTP strict transport security). Also when I do lookups to public SSL verification sites they say there are no HPKP or HSTS headers being used on that site.

Why would this be ?

Could it be to do with the fact that some sites aggregate content from many other sites and perhaps one of those sites is using HPKP or HSTS headers ?

The other strange issue is that sometime these problem sites work for users and sometimes they don't. Could this related to the above, in that these sites may be dynamically and variably pulling content from third party sites that use HPKP or HSTS headers ?

Has anyone else encountered this sort of issue ?

Thanks kindly.

[hmtay_FTNT · Accepted solution]

Hi fran1942,

>>I am finding instances of SSL certificate pinning (HPKP) where I need to make exceptions to the DPI list e.g. *.google.com etc.

Did you use a self-signed certificate or a certificate signed by a trusted third party CA? You would also get different results with different browsers. For example, Mozilla I believe in one of the release have the security.cert_pinning.enforcement_level set to 2 by default. However, since then, they have set it back to 1 which is the same as the default Chrome settings. The current default settings for both Chrome and Firefox allows MiTM if the trust anchor is a self-signed CA installed by the user.

https://wiki.mozilla.org/SecurityEngineering/Public_Key_Pinning

https://www.chromium.org/Home/chromium-security/security-faq#TOC-How-does-key-pinning-interact-with-...

In both browsers too, only very little sites do certificate pinning. This is a key point with browsers vs stand alone applications. Due to the fact that there are too many HTTPS sites that can be accessed, it is not possible for browsers to pin all the certificates to the browser. Thus, you are unlikely to see much certificate pinning errors with a browser. Only certain sites like the Google sites, Twitter, etc have the keys hardcoded in the browser list. With a standalone application, certificate pinning is easily do-able since it does not have to cover such a big range of sites. 

The browser settings could be one reason why some sites work sometimes and not other times. In any case, if you are using a self-signed certificate and you have imported the certificate into the trusted root CA list, you should not run into any problems on the browsers unless you are using Firefox and have the cert_pinning.enforcement_level set to 2. On my environment, I have no problem doing deep-inspection on Google sites. 

Let me know if you have more questions! Thanks.

HoMing

[hmtay_FTNT · Accepted solution]

Hi fran1942,

>>I am finding instances of SSL certificate pinning (HPKP) where I need to make exceptions to the DPI list e.g. *.google.com etc.

Did you use a self-signed certificate or a certificate signed by a trusted third party CA? You would also get different results with different browsers. For example, Mozilla I believe in one of the release have the security.cert_pinning.enforcement_level set to 2 by default. However, since then, they have set it back to 1 which is the same as the default Chrome settings. The current default settings for both Chrome and Firefox allows MiTM if the trust anchor is a self-signed CA installed by the user.

https://wiki.mozilla.org/SecurityEngineering/Public_Key_Pinning

https://www.chromium.org/Home/chromium-security/security-faq#TOC-How-does-key-pinning-interact-with-...

In both browsers too, only very little sites do certificate pinning. This is a key point with browsers vs stand alone applications. Due to the fact that there are too many HTTPS sites that can be accessed, it is not possible for browsers to pin all the certificates to the browser. Thus, you are unlikely to see much certificate pinning errors with a browser. Only certain sites like the Google sites, Twitter, etc have the keys hardcoded in the browser list. With a standalone application, certificate pinning is easily do-able since it does not have to cover such a big range of sites. 

The browser settings could be one reason why some sites work sometimes and not other times. In any case, if you are using a self-signed certificate and you have imported the certificate into the trusted root CA list, you should not run into any problems on the browsers unless you are using Firefox and have the cert_pinning.enforcement_level set to 2. On my environment, I have no problem doing deep-inspection on Google sites. 

Let me know if you have more questions! Thanks.

HoMing

[fran1942]

Hello, thank you for that information.

So to summarise, what sort of things apart from key pinning can cause deep packet inspection to fail ?

n.b. I do have a self-signed cert which we have distributed to all computers in our company.

[fran1942]

...additionally, how do I completely disable pinning in Chrome, because unless I make a DPI exception, then Google sites definitely do fail e.g. Google Drive.

[hmtay_FTNT]

The major cause of failure for deep-inspection is certificate pinning. Aside from that, from time to time, when Chrome tries a new cipher suite, we need to update the engine to cover those new suites. 

If you imported your self-signed certificate on all your devices, you should not have any problems with any sites on Chrome. When the Google sites failed, what was the error? Did you block "QUIC" on Chrome? Can I know whick FortiOS and IPS Engine are you using?

[fran1942]

Hello, I am using FortiOS version 5.4.1.

Sorry, I don't understand when you say 'if you imported your self-signed certificate on all your devices, you should not have any problems with any sites on Chrome'.

From what I understand Chrome does have built in certificate pinning for certain sites and it also receives HPKP headers from websites which can specify pinning requirements. I understand this is regardless of whether or not you have your self-signed certificate installed on the user machine or not.

How does having your self-signed cert installed bypass all pinning via HPKP ?

Thank you for any information.

[hmtay_FTNT · Accepted solution]

>>From what I understand Chrome does have built in certificate pinning for certain sites and it also receives HPKP headers from websites which can specify pinning requirements. I understand this is regardless of whether or not you have your self-signed certificate installed on the user machine or not.

If you read the link I sent you earlier on Chrome, here's what it says:

Chrome does not perform pin validation when the certificate chain chains up to a private trust anchor. A key result of this policy is that private trust anchors can be used to proxy (or MITM) connections, even to pinned sites. “Data loss prevention” appliances, firewalls, content filters, and malware can use this feature to defeat the protections of key pinning.We deem this acceptable because the proxy or MITM can only be effective if the client machine has already been configured to trust the proxy’s issuing certificate — that is, the client is already under the control of the person who controls the proxy (e.g. the enterprise’s IT administrator). If the client does not trust the private trust anchor, the proxy’s attempt to mediate the connection will fail as it should. What this means is if your self-signed certificate is placed in the trusted root CA list, Chrome will not perform pin validation both for certificate pinning and HPKP. This is the default Chrome setting and also the default Firefox setting as of the current version with cert_pinning.enforcement_level set to 1.  https://developer.mozilla.org/en-US/docs/Web/HTTP/Public_Key_Pinning https://bugzilla.mozilla.org/show_bug.cgi?id=1168603