No fortiguard, forticloud, forticare on branch FGT

[Tutek]

Hi,

I have problem on the branch fortigate it route all traffic to hq fortigate, and I cannot run fortiguard (Unable to connect to FortiGuard servers.) Forticloud - failed to load data etc.

In the branch console I can ping all fortiguard servers, I can telnet to 514 ports like:

execute telnet 173.243.132.171 514
Trying 173.243.132.171...
Connected to 173.243.132.171.

on hq fortigate I see in logs that traffic from branch destined to fortiguard like:

13.248.131.62
173.243.132.27

are allowed and go out to the internet but these service are not working, in log I have something like:

[3038] fds_download_image_list: TRACE
[41] fds_queue_task: req-1 is added to fds-update
[579] fds_https_start_server: server: 13.248.131.62:443
[580] fds_https_start_server: source-ip: 0.0.0.0:0
[113] __ssl_cert_ctx_load: Added cert /etc/cert/factory/root_Fortinet_Factory.cer, root ca Fortinet_CA, idx 0 (default)
[480] ssl_ctx_use_builtin_store: Loaded Fortinet Trusted Certs
[486] ssl_ctx_use_builtin_store: Enable CRL checking.
[493] ssl_ctx_use_builtin_store: Enable OCSP Stapling.
[755] ssl_ctx_create_new_ex: SSL CTX is created
[782] ssl_new: SSL object is created
[82] https_create: proxy server 0.0.0.0 port:0
[185] forticldd_add_hostname_check: Add hostname checking 'globalupdate2.fortinet.net'
[359] __ssl_crl_verify_cb: Cert error 20, unable to get local issuer certificate. Depth 0
[1894] SSL_dump_handshake_err: Certificate failed verification. Error: 20 (unable to get local issuer certificate), depth: 0, subject: /jurisdictionC=US/jurisdictionST=Delaware/businessCategory=Private Organization/serialNumber=3321792/C=US/ST=California/L=Sunnyvale/O=Fortinet, Inc./CN=globalupdate2.fortinet.net.
[1001] ssl_connect: SSL_connect failes: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
[477] fds_https_connect: https_connect(13.248.131.62:443) failed: ssl_connect() failed: 0 (error:00000000:lib(0):func(0):reason(0)).
[203] __ssl_data_ctx_free: Done
[1046] ssl_free: Done
[195] __ssl_cert_ctx_free: Done
[1056] ssl_ctx_free: Done
[1037] ssl_disconnect: Shutdown
[238] fds_svr_default_on_error: fds-update: ip=13.248.131.62:443, reason=4
[255] fds_svr_default_on_error: fds-update: Conn failes 1/1
[275] fds_svr_default_on_error: fds-update: req-id=1, num_try=1, read=0, reason=4
[2993] tsk_send_image_list: num=-1
[465] fds_send_reply: Sending 0 bytes data.
[421] fds_free_tsk: cmd=1; req.noreply=1
[421] fds_free_tsk: cmd=1; req.noreply=0
[186] fds_svr_default_task_xmit: try to get IPs for fds-update
[254] fds_resolv_addr: resolve 'globalupdate2.fortinet.net'
[186] fds_get_addr: name=globalupdate2.fortinet.net, id=14041, cb=0x9588e8
[102] dns_parse_resp: DNS globalupdate2.fortinet.net -> 13.248.131.62
[102] dns_parse_resp: DNS globalupdate2.fortinet.net -> 76.223.2.16
[137] fds_svr_default_pickup_server: fds-update: 13.248.131.62:443
[3274] fds_handle_request: Received cmd 116 from pid-1298, len 0
[465] fds_send_reply: Sending 8 bytes data.
[3274] fds_handle_request: Received cmd 116 from pid-1298, len 0
[465] fds_send_reply: Sending 8 bytes data.
[3274] fds_handle_request: Received cmd 101 from pid-1298, len 0
[41] fds_queue_task: req-101 is added to message-controller
[579] fds_https_start_server: server: 173.243.132.27:443
[580] fds_https_start_server: source-ip: 0.0.0.0:0
[113] __ssl_cert_ctx_load: Added cert /etc/cert/factory/root_Fortinet_Factory.cer, root ca Fortinet_CA, idx 0 (default)
[480] ssl_ctx_use_builtin_store: Loaded Fortinet Trusted Certs
[486] ssl_ctx_use_builtin_store: Enable CRL checking.
[493] ssl_ctx_use_builtin_store: Enable OCSP Stapling.
[755] ssl_ctx_create_new_ex: SSL CTX is created
[782] ssl_new: SSL object is created
[82] https_create: proxy server 0.0.0.0 port:0
[185] forticldd_add_hostname_check: Add hostname checking 'globalmsgctrl2.fortinet.net'
[359] __ssl_crl_verify_cb: Cert error 20, unable to get local issuer certificate. Depth 0
[1894] SSL_dump_handshake_err: Certificate failed verification. Error: 20 (unable to get local issuer certificate), depth: 0, subject: /jurisdictionC=US/jurisdictionST=Delaware/businessCategory=Private Organization/serialNumber=3321792/C=US/ST=California/L=Sunnyvale/O=Fortinet, Inc./CN=globalctrl.fortinet.net.
[1001] ssl_connect: SSL_connect failes: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
[477] fds_https_connect: https_connect(173.243.132.27:443) failed: ssl_connect() failed: 0 (error:00000000:lib(0):func(0):reason(0)).
[203] __ssl_data_ctx_free: Done
[1046] ssl_free: Done
[195] __ssl_cert_ctx_free: Done
[1056] ssl_ctx_free: Done
[1037] ssl_disconnect: Shutdown

upd_daemon[1782]-Received update now request
upd_daemon[1508]-Found cached action=00000002
do_update[492]-Starting now UPDATE (final try)
upd_fds_load_default_server6[1105]-Resolve fds ipv6 address failed.
upd_comm_connect_fds[458]-Trying FDS 76.223.2.16:443
tcp_connect_fds[234]-Binding to interface 122
[113] __ssl_cert_ctx_load: Added cert /etc/cert/factory/root_Fortinet_Factory.cer, root ca Fortinet_CA, idx 0 (default)
[480] ssl_ctx_use_builtin_store: Loaded Fortinet Trusted Certs
[486] ssl_ctx_use_builtin_store: Enable CRL checking.
[493] ssl_ctx_use_builtin_store: Enable OCSP Stapling.
[755] ssl_ctx_create_new_ex: SSL CTX is created
[782] ssl_new: SSL object is created
[166] ssl_add_ftgd_hostname_check: Add hostname checking 'globalupdate2.fortinet.net'
[359] __ssl_crl_verify_cb: Cert error 20, unable to get local issuer certificate. Depth 0
__upd_peer_vfy[331]-Server certificate failed verification. Error: 20 (unable to get local issuer certificate), depth: 0, subject: /jurisdictionC=US/jurisdictionST=Delaware/businessCategory=Private Organization/serialNumber=3321792/C=US/ST=California/L=Sunnyvale/O=Fortinet, Inc./CN=globalupdate2.fortinet.net.
[1001] ssl_connect: SSL_connect failes: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
ssl_connect_fds[392]-Failed SSL connecting (5,0,Success)
[203] __ssl_data_ctx_free: Done
[1046] ssl_free: Done
[195] __ssl_cert_ctx_free: Done
[1056] ssl_ctx_free: Done
upd_comm_connect_fds[476]-Failed SSL connect
upd_comm_connect_fds[458]-Trying FDS 13.248.131.62:443
tcp_connect_fds[234]-Binding to interface 122
[113] __ssl_cert_ctx_load: Added cert /etc/cert/factory/root_Fortinet_Factory.cer, root ca Fortinet_CA, idx 0 (default)
[480] ssl_ctx_use_builtin_store: Loaded Fortinet Trusted Certs
[486] ssl_ctx_use_builtin_store: Enable CRL checking.
[493] ssl_ctx_use_builtin_store: Enable OCSP Stapling.
[755] ssl_ctx_create_new_ex: SSL CTX is created
[782] ssl_new: SSL object is created
[166] ssl_add_ftgd_hostname_check: Add hostname checking 'globalupdate2.fortinet.net'
[359] __ssl_crl_verify_cb: Cert error 20, unable to get local issuer certificate. Depth 0
__upd_peer_vfy[331]-Server certificate failed verification. Error: 20 (unable to get local issuer certificate), depth: 0, subject: /jurisdictionC=US/jurisdictionST=Delaware/businessCategory=Private Organization/serialNumber=3321792/C=US/ST=California/L=Sunnyvale/O=Fortinet, Inc./CN=globalupdate2.fortinet.net.
[1001] ssl_connect: SSL_connect failes: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
ssl_connect_fds[392]-Failed SSL connecting (5,0,Success)
[203] __ssl_data_ctx_free: Done
[1046] ssl_free: Done
[195] __ssl_cert_ctx_free: Done
[1056] ssl_ctx_free: Done
upd_comm_connect_fds[476]-Failed SSL connect
do_update[504]-UPDATE failed
do_check_wanip[655]-Starting getting wan ip
upd_fds_load_default_server6[1105]-Resolve fds ipv6 address failed.
upd_comm_connect_fds[458]-Trying FDS 76.223.2.16:443
tcp_connect_fds[234]-Binding to interface 122
[113] __ssl_cert_ctx_load: Added cert /etc/cert/factory/root_Fortinet_Factory.cer, root ca Fortinet_CA, idx 0 (default)
[480] ssl_ctx_use_builtin_store: Loaded Fortinet Trusted Certs
[486] ssl_ctx_use_builtin_store: Enable CRL checking.
[493] ssl_ctx_use_builtin_store: Enable OCSP Stapling.
[755] ssl_ctx_create_new_ex: SSL CTX is created
[782] ssl_new: SSL object is created
[166] ssl_add_ftgd_hostname_check: Add hostname checking 'globalupdate2.fortinet.net'
[359] __ssl_crl_verify_cb: Cert error 20, unable to get local issuer certificate. Depth 0
__upd_peer_vfy[331]-Server certificate failed verification. Error: 20 (unable to get local issuer certificate), depth: 0, subject: /jurisdictionC=US/jurisdictionST=Delaware/businessCategory=Private Organization/serialNumber=3321792/C=US/ST=California/L=Sunnyvale/O=Fortinet, Inc./CN=globalupdate2.fortinet.net.
[1001] ssl_connect: SSL_connect failes: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
ssl_connect_fds[392]-Failed SSL connecting (5,0,Success)
[203] __ssl_data_ctx_free: Done
[1046] ssl_free: Done
[195] __ssl_cert_ctx_free: Done
[1056] ssl_ctx_free: Done
upd_comm_connect_fds[476]-Failed SSL connect
upd_comm_connect_fds[458]-Trying FDS 13.248.131.62:443
tcp_connect_fds[234]-Binding to interface 122
[113] __ssl_cert_ctx_load: Added cert /etc/cert/factory/root_Fortinet_Factory.cer, root ca Fortinet_CA, idx 0 (default)
[480] ssl_ctx_use_builtin_store: Loaded Fortinet Trusted Certs
[486] ssl_ctx_use_builtin_store: Enable CRL checking.
[493] ssl_ctx_use_builtin_store: Enable OCSP Stapling.
[755] ssl_ctx_create_new_ex: SSL CTX is created
[782] ssl_new: SSL object is created
[166] ssl_add_ftgd_hostname_check: Add hostname checking 'globalupdate2.fortinet.net'
[359] __ssl_crl_verify_cb: Cert error 20, unable to get local issuer certificate. Depth 0
__upd_peer_vfy[331]-Server certificate failed verification. Error: 20 (unable to get local issuer certificate), depth: 0, subject: /jurisdictionC=US/jurisdictionST=Delaware/businessCategory=Private Organization/serialNumber=3321792/C=US/ST=California/L=Sunnyvale/O=Fortinet, Inc./CN=globalupdate2.fortinet.net.
[1001] ssl_connect: SSL_connect failes: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
ssl_connect_fds[392]-Failed SSL connecting (5,0,Success)
[203] __ssl_data_ctx_free: Done
[1046] ssl_free: Done
[195] __ssl_cert_ctx_free: Done
[1056] ssl_ctx_free: Done
upd_comm_connect_fds[476]-Failed SSL connect
do_check_wanip[659]-Failed getting wan ip

something with certificates I see but how to fix this? On branch I don't use deep ssl scan, only at hq when going to internet.

[Tutek · Accepted solution]

OK, problem resolved.

I have created a rule to internet services  - fortiguard and moved it to the top of rules, this way traffic to fortiguard from branch is not doing ssl scan.