SSL Inspection blocks Windows Update despite exemptions (Windows 10),

[x_member]

FortiOS 5.2.11 on FGT 60D

I'm attempting to re-introduce deep inspection of outbound traffic from laptops used on our network following a long standing ssl inspection issue being patched.

The ssl inspection configuration and exemptions we had in place prior to the bug allowed laptops to run Windows Update without issue.

Currently on my laptop (Windows 10 Pro) Windows Update fails with error 0x80240437 when run from behind the Fortigate; if I run from within our guest network (which is not behind a Fortigate / SSL inspection) it completes successfully.

I can see in the WindowsUpdate.log that this is (probably) down to a certificate check failure:

2017/07/19 10:36:59.4330139 16176 14616 IdleTimer [0]3F30.3918::07/19/2017-10:36:59.433 [agent]WU operation (CAgentProtocolTalker::SyncUpdates_WithRecover) started; operation # 10; does<NULL> use network; is<NULL> at background priority<NULL>
2017/07/19 10:36:59.4330826 16176 14616 WebServices [0]3F30.3918::07/19/2017-10:36:59.433 [webserviceinfra]Auto proxy settings for this web service call.
2017/07/19 10:37:00.4889480 16176 14616 WebServices [0]3F30.3918::07/19/2017-10:37:00.488 [client]Certificate failed SSL intermediate CA check.
2017/07/19 10:37:00.4890632 16176 14616 WebServices [0]3F30.3918::07/19/2017-10:37:00.489 [webserviceinfra]WS error: There was an error communicating with the endpoint at 'https://fe3.delivery.mp.microsoft.com/ClientWebService/client.asmx'.
2017/07/19 10:37:00.4890703 16176 14616 WebServices [0]3F30.3918::07/19/2017-10:37:00.489 [webserviceinfra]Web service call failed with hr = 80240437.

I've spent numerous hours trying to resolve this, however I cannot see what I am missing despite an ever expanding list of exemptions under my "WindowsUpdate" address group:

config firewall ssl-ssh-profile
    edit "deep-inspection"
        set comment "Deep inspection."
            config https
                set ports 443
            end
            config ftps
                set ports 990
            end
            config imaps
                set ports 993
            end
            config pop3s
                set ports 995
                set status disable
            end
            config smtps
                set ports 465
            end
            config ssl-exempt
                <snipped for brevity>
                edit 27
                    set type address
                    set address "WindowsUpdate"
            end
        set caname "DPI"
        set ssl-invalid-server-cert-log enable
    end 
 
config firewall addrgrp
    edit "WindowsUpdate"
        set uuid 38db89c2-e371-51e4-1f5b-c23edb9bdf46
        set member "*.download.windowsupdate.com" "*.update.microsoft.com" "*.windowsupdate.com" "*.windowsupdate.microsoft.com" "download.microsoft.com" "download.windowsupdate.com" "ds.download.windowsupdate.com" "msftncsi.com" "ntservicepack.microsoft.com" "stats.update.microsoft.com" "test.stats.update.microsoft.com" "update.microsoft.com" "windowsupdate.microsoft.com" "wustat.windows.com" "crl.microsoft.com" "ctldl.windowsupdate.com" "au.download.windowsupdate.com" "fe2.update.microsoft.com" "AkamaiContentDelivery" "delivery_mp_microsoft_com" "ws_microsoft_com" "fe3_update_microsoft_com" "sls_microsoft_com"
    end 

For servers we use a separate application control policy to allow updates that is only enabled during maintenance windows; laptop web traffic is not restricted, we simply wish to scan HTTPS for IPS /  AV purposes.

My guess is that some additional mechanisms have been introduced by Microsoft since we last had this policy enabled for laptops - I'd appreciate any pointers from the community at this stage.

[x_member]

For completeness, the address group members:

config firewall address
 edit "*.download.windowsupdate.com"
        set uuid d38ab5ee-e2b6-51e4-6811-7ee23f1e5878
        set type fqdn
        set fqdn "*.download.windowsupdate.com"
    next
    edit "*.update.microsoft.com"
        set uuid 7ed92d50-e2b6-51e4-0a3d-207abf35eb4d
        set type fqdn
        set fqdn "*.update.microsoft.com"
    next
 edit "*.windowsupdate.com"
        set uuid 8f997938-e2b6-51e4-ebad-5f46365f3ab4
        set type fqdn
        set fqdn "*.windowsupdate.com"
    next
 edit "*.windowsupdate.microsoft.com"
        set uuid 66ebdab2-e2b6-51e4-0205-d64a7fc3836a
        set type fqdn
        set fqdn "*.windowsupdate.microsoft.com"
        set uuid bda89f7a-e2b6-51e4-ac56-3a6fe6fa1547
    next
 edit "download.microsoft.com"
        set type fqdn
        set fqdn "download.microsoft.com"
    next
 edit "download.windowsupdate.com"
        set uuid a04589b6-e2b6-51e4-8380-45c0c9fb0079
        set type fqdn
        set fqdn "download.windowsupdate.com"
    next
    edit "ds.download.windowsupdate.com"
        set uuid 3eb02cd2-e361-51e4-9f9d-f30206ea75e1
        set type fqdn
        set fqdn "ds.download.windowsupdate.com"
    next
    edit "msftncsi.com"
        set uuid 1e7ec04a-e2b7-51e4-fab0-ee96a6805e60
        set type fqdn
        set fqdn "msftncsi.com"
    next
    edit "ntservicepack.microsoft.com"
        set uuid 0b511536-e2b7-51e4-c03e-fe226297dac7
        set type fqdn
        set fqdn "ntservicepack.microsoft.com"
    next
    edit "stats.update.microsoft.com"
        set uuid f4afa0d6-e2b6-51e4-2c03-d7c1c23a31a0
        set type fqdn
        set fqdn "stats.update.microsoft.com"
    next
    edit "test.stats.update.microsoft.com"
        set uuid 14ac0860-e358-51e4-fee2-a9d3c2029924
        set type fqdn
        set fqdn "test.stats.update.microsoft.com"
    next
    edit "update.microsoft.com"
        set uuid b1948a3a-9ffb-51e4-f60b-81a1028a1e41
        set type fqdn
        set fqdn "update.microsoft.com"
    next
    edit "windowsupdate.microsoft.com"
        set uuid 53e9615a-e2b6-51e4-0a82-3dff7b063060
        set type fqdn
        set fqdn "windowsupdate.microsoft.com"
    next
    edit "wustat.windows.com"
        set uuid 3118405a-e2b7-51e4-2db2-7282d055c0b5
        set type fqdn
        set fqdn "wustat.windows.com"
    next
    edit "crl.microsoft.com"
        set uuid 3fffce5e-e379-51e4-1a51-784f8b01b019
        set type fqdn
        set fqdn "crl.microsoft.com"
    next
    edit "ctldl.windowsupdate.com"
        set uuid 4ff14cde-e379-51e4-d24e-a52781fd37b5
        set type fqdn
        set fqdn "ctldl.windowsupdate.com"
    next
    edit "au.download.windowsupdate.com"
        set uuid 2dd995fc-e379-51e4-59dd-5842c00704a4
        set type fqdn
        set fqdn "au.download.windowsupdate.com"
    next
    edit "fe2.update.microsoft.com"
        set uuid 0037da80-fc8a-51e4-e1b9-eaef59739967
        set type fqdn
        set fqdn "fe2.update.microsoft.com"
    next
    edit "AkamaiContentDelivery"
        set uuid 220095f8-2c5a-51e5-797a-63c7cee9ab2e
        set member "AkamaiContent1" "AkamaiContent2"
        set comment "Akamai content delivery network"
    next
    edit "AkamaiContent1"
        set uuid ebe86900-2c59-51e5-b90f-0a21d8b4f988
        set type fqdn
        set comment "Akamai content delivery network"
        set associated-interface "wan1"
        set fqdn "*.deploy.akamaitechnologies.com"
    next
    edit "AkamaiContent2"
        set uuid 03d551f4-2c5a-51e5-f53e-866580e98268
        set type fqdn
        set comment "Akamai content delivery network"
        set associated-interface "wan1"
        set fqdn "*.deploy.static.akamaitechnologies.com"
    next
    edit "delivery_mp_microsoft_com"
        set uuid ac65c496-6b9d-51e7-cd20-8932289339c0
        set type fqdn
        set fqdn "*.delivery.mp.microsoft.com"
    next
    edit "ws_microsoft_com"
        set uuid ea0fef4c-6bcf-51e7-35de-8f16af209be7
        set type fqdn
        set fqdn "*.ws.microsoft.com"
    next
    edit "fe3_update_microsoft_com"
        set uuid 9d42cebe-6c56-51e7-dadf-f7be7034f1a7
        set type fqdn
        set fqdn "fe3.update.microsoft.com"
    next
    edit "sls_microsoft_com"
        set uuid 83a23cce-6c65-51e7-e0cd-88664b237127
        set type fqdn
        set fqdn "*.sls.microsoft.com"
    end 

[hmtay_FTNT · Accepted solution]

Hello CodeMonkey,

Can you do a packet capture with the settings you have right now? I can identify which session was blocked by the IPS Engine for you. We just have to look for sessions where the Fortigate replaces the SSL Certificate. You can send it to me at my email at hmtay@fortinet.com

Homing

[x_member]

hmtay wrote:

Hello CodeMonkey,

Can you do a packet capture with the settings you have right now? I can identify which session was blocked by the IPS Engine for you. We just have to look for sessions where the Fortigate replaces the SSL Certificate. You can send it to me at my email at hmtay@fortinet.com

Homing

Hi Homing, 

Thanks for the offer, but I now have a ticket open with support to help me resolve this. 

I think it's probably better I proceed with debugging via the ticket.

I'll update this thread with the solution once resolved.

[x_member]

In the end it appears that there was a simple solution to this - one that I'm told now ships as part of the default configuration.

I had to add a wildcard FQDN object for *.microsoft.com and include it in the exemptions.

This in turn enabled me to get rid of some redundant FQDNs from exemptions:

fe2.update.microsoft.com

*.deploy.akamaitechnologies.com *.deploy.static.akamaitechnologies.com *.delivery.mp.microsoft.com

*.ws.microsoft.com fe3.update.microsoft.com *.sls.microsoft.com

All seems well with the revised configuration having tested against Windows Update on multiple laptops (Windows 10, Windows 8 Pro).

[ergotherego]

Hi CodeMonkey,

Is this working for you? I am trying to do something similar. I want to enable outbound HTTP/HTTPS traffic to Windows Update, but block other traffic.

I tried using FQDN records but due to CDN the firewall and servers end up getting different IP addresses sometimes - even when using the same DNS resolver.

I do see you have a few names in your list that I don't have in mine, so was curious if it's been stable for you using that addrgrp you have. Also, how did you come to know those Akamai records? I haven't seen those in postings about Windows Update domain names.

Thanks,

[x_member]

Hi ergotherego,

Yes, I have Windows Update working successfully in the scenario covered in this thread. 

Regarding the additional names I have - they may not all be relevant at this time but they seemed to be when I first looked to install our Fortigates a couple of years ago. Things have obviously moved on since then hence my need to post on this subject.

Regarding your situation I'm not really sure what to suggest, as we allow all web traffic for the laptops (excluding known bad actors using web filters etc.). Perhaps a separate policy using application control might be more appropriate for your needs? We use this approach for windows updates on servers, with specific policies permitting Windows Updates traffic enabled during maintenance windows only.