Description
This article describes digital certificates and explains the use and validation of them.
Certificates come with the use of the Secure Sockets Layer (SSL) or its successor, Transport Layer Security (TLS, latest version 1.3).
Certificates are required for use with SSL/TLS and so certain standard handling needs to be applied.
This article also gives a few examples of file types to care for when handling certificates.
Scope
Digital certificates are typically used to build up SSL/TLS tunnels in various products, FortiClient EMS, FortiAnalyzer, FortiGate, FortiNAC, FortiAuthenticator, FortiConnect, FortiWeb, and many more.
All of those use certificates already with the web server so where the graphical user interface is available.
Additional functions like Agents or client applications may also make use of certificates.
A small and incomplete list:
- FortiClient VPN.
- FortiClient Mobility Agent.
- Fortinet FSSO Collector Agent.
- FortiNAC Persistent Agent.
- FortiGate <> FortiClientEMS.
Solution
Certificates generally will be used for encrypting a payload.
The origin of the payload typically does not matter.
One indicator is the 'S' at the end of some protocols. SMTP(S), LDAP(S), and FTP(S) are such examples.
These protocols include (but are not limited to):
- LDAPS.
- SMTPS (and a compatibility idea called 'STARTTLS').
- FTPS (not to be mixed up with SFTP which is FTP over SSH).
- HTTPS.
- POP3S.
- IMAPS.
- SIPS.
- XMPPS.
These protocols function by creating a TLS tunnel prior to sending or transporting any protocol respective payload.
If the tunnel cannot be established, the payload will not be sent. Troubleshooting missing, incorrect data when the tunnel cannot be established is not necessary. Likewise, when traffic/payload is noted, the tunnel likely has been established.
Meanwhile, RADIUS' extensible authentication protocol (EAP) can significantly extend functionality and security. EAP methods that use TLS include EAP-TLS, EAP-TTLS, EAP-PEAP. The secure RADIUS implementation known as RADSEC, rfc6614, also uses TLS.
Since TLS works as a tunnel in which payload is transferred, VPN solutions based on SSL or TLS supersede SSL.
These include OpenVPN and SSLVPN. Note that these are not compatible with each other as each implementation can be configured differently.
There are several occasions in which certificates are not only used by a server also used by a client who is using a certificate to authenticate to a server.
All of these certificate exchanges have one thing in common:
- A certificate will be exchanged from node A to another node B.
- Node B will verify whether the certificate is valid or not.
Note:
It does not matter which protocol triggers this certificate exchange, or which program is using TLS for the exchange.
Neither does the TLS version matter, nor does it matter which setup is used or in which environment.
It may be the case that multiple certificates are exchanged from client to server and from server to client (which is the typical direction).
If SSL/TLS is used, there will be a certificate exchange.
Some implementations allow skipping the certificate validation (after the exchange). Otherwise, there is no way to bypass this.
One prominent example is a certificate warning in a browser when attempting to connect to a website that a browser fails to validate the certificate for.
If 'advanced options' are used, the user can connect anyway: the user has instructed the browser to skip the validation for the website in that instance and in the future.
As a TLS client, FortiGate also allows for LDAPS to not verify the certificate chain of the certificate sent by the LDAPS server.
Once the TLS tunnel is established, traffic/payload flows between the nodes. In a packet capture, only encrypted traffic will be seen.
This also means that, if encrypted traffic is seen, the tunnel is up and the certificate exchange has succeeded.
The certificate exchange can be monitored through various methods.
- If TLS <v1.3 is used, Wireshark shows the certificate exchange directly. With TLS1.3, the certificate exchange will not be visible anymore.
- In either cases, logs are visible from both ends. The corresponding software should generate appropriate logs, which will be best observed when trying to establish such a tunnel. imestamps are crucial to understanding the exchange.
In the case of software that is installed on an OS, the logs will have defined locations.
- With Windows, the logs are often located in the installation directory of the product, but the documentation of the software may provide more insight.
- With Linux, part of the logs may be included in a file in the directory or subdirectory at /var/log/ or in a directory specified by the product documentation.
The exchange should be logged or captured similarly to:
- TLS Client -> Client Hello -> TLS Server.
Client Hello:
- Contains support ciphers.
2. TLS Server -> Server Hello -> TLS Client.
Server hello:
- Contains chosen cipher (overlaps with the client support ciphers).
- Contains server details.
- Contains a server certificate.
- MAY contain a request for a client certificate if configured on the server.
3. TLS Client -> Client Key Exchange -> TLS Server.
- The exchange will lead to a change of cipher spec (encrypted).
- If requested from the server, this message will also contain a certificate from the client.
Note:
As specified earlier, the certificate exchange is not visible after the client 'hello' in TLS1.3. The exchange is helpful for troubleshooting. For test cases with OpenSSL, it can enforce the use of TLS1_2 (with the option -tls1_2).
4. Change the Cipher Spec again. After, the TLS tunnel will be up.
The next step is validation of the certificate.
Keep in mind that one node sends a certificate and the other Node's job is to verify it.
Verification is mainly done against:
1. Certificate chain.
2. Certificate validity period (whole chain).
3. Certificate revocation information.
The listed methods in detail:
1. Certificate Chain:
A certificate chain is a link of an end entity certificate (web server certificate or client certificate for example) to another certificate that issued this same certificate. This issuing certificate usually goes by the name of Certificate Authority (CA).
With the example of a server certificate, a chain typically looks like this:
- Server certificate.
- Server certificate is issued by an intermediate CA certificate, also called a subordinate CA.
- A subordinate CA is issued by a root CA.
In other words: The root CA issues an intermediate CA certificate. The intermediate CA certificate issues a server certificate.
More detailed: The 'issuing' is actually the process of signing a public key.
A certificate consists of a public and a private key. This is called a key pair.
The keys belong to each other and cannot be exchanged.
This is a feature that makes certificates and key pairs secure to use as the key cannot be faked.
The two keys are:
- A public key that is available to everybody. In some documentation, this is abbreviated as 'pubkey'.
- A private key that stays with this entity only (for example: a web server, or an end user as a client). In some documentation, this is abbreviated as 'privkey'.
Uniquely, data encrypted with one of these two keys can be only decrypted with the other key.
Data encrypted with the private key can be decrypted with the fitting public key.
Data encrypted with the public key can be decrypted with the fitting private key.
The keys can be copied, but a fake key to imitate the private key fitting this keypair cannot be easily created. Note: Usage of private and public keys applies to the SSH protocol as well.
The illustration above shows the actions of the private key signing the other certificates' public key.
It encrypts data on the public key of the certificate next in the chain.
On the other side are the signatures, meaning the encrypted data is decrypted/verified with the public key belonging to the key pair that created the encrypted data/signature.
The signature decryption with the public key can only work if the public key is available.
The strength of a key will indicate how safe it is, making it more difficult to break security by faking a key in order to decrypt data with a key that does not belong to this key pair.
Typical key lengths are RSA 2048bit, but an increasing number of keys now use RSA 4096.
Additionally, ECDSA keys have a much smaller size such as 256 bit.
The root CA will sign the intermediate key pair's public key using its private key.
This signature is a data block that is encrypted with the private key of the signer.
It can be decrypted with the public key of the signer. Another public key cannot be used to decrypt the data.
The public key is required to verify that a certificate was signed (or issued) by a specific authority certificate.
The public keys of certificates are distributed to clients in the 'Trusted root certificate authority store'.
Web browsers and operating systems have a pre-filled store with the public keys of the known certificate authorities.
If an error message is presented for an 'unknown CA', it means that the verification of the chain failed as the client is missing a public key in the certificate chain.
The log entries vary on implementation, product and scenario, but some examples may look like the following cases.
- LS-Server missing the clients' issuing CA:
eap_tls: (TLS) OpenSSL says error 20 : unable to get local issuer certificate
eap_tls: (TLS) send TLS 1.2 Alert, fatal unknown_ca
rest_reject: Value : "eap_tls: (TLS) OpenSSL says error 20 : unable to get local issuer certificate"
rest_reject: Value : "eap_tls: (TLS) Alert write:fatal:unknown CA"
eap_tls: ERROR: (TLS) Alert read:fatal:unknown CA
eap_tls: (TLS) The client is informing us that it does not recognize the CA used to issue the server certificate. Please update the client so that it knows about the CA.
recv TLS 1.2 Alert, fatal unknown_ca
eap_peap: ERROR: (TLS) Alert read:fatal:unknown CA
eap_peap: ERROR: (TLS) Failed reading application data from OpenSSL: error:0A000418:SSL routines::tlsv1 alert unknown ca
- TLS-Client missing the servers issuing CA:
eap_tls: ERROR: (TLS) OpenSSL says error 2 : unable to get issuer certificate
eap_tls: ERROR: (TLS) Alert write:fatal:unknown CA
Server Certificate Not Trusted (Unknown CA)
SSL Alert read: fatal unknown CA
Error error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
SSL accept failed. SSL_accept()=-1 SSL_get_error()=6
__verify_cb-Cert error 20, unable to get local issuer certificate. Depth 0. Subject ''
__ldap_tcps_connect-tcps_connect(192.168.48.95) failed: ssl_connect() failed: 167772294 (error:0A000086:SSL routines::certificate verify failed)
This could be captured in either case, packet capture read with WireShark):
TLSv1.3 Record Layer: Alert (Level: Fatal, Description: Unknown CA)
TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Unknown CA)
Web servers typically send their server certificate and all intermediate certificates up to, but not including, the root CA public key. Some may additionally send the root CA certificate with them.
In both cases:
- The root CA must be installed on the client as a 'Trusted root certificate authority'.
- The receiving node must be able to complete the chain of certificates.
- The receiving node, TLS client, must find the FQDN it is addressing in the 'subject' and in many cases, in the 'Subject Alternative Name' (SAN). Wildcard certificates that specify a * to match any FQDN are usually accepted (such as *.forti.lab, which would match www.forti.lab as an example), exceptions are Client certificates and RADIUS server certificates.
Note:
The chain may consist of a minimum of two certificates, a root CA and a server certificate. It may additionally contain a root CA, a number of intermediate CAs, and a server certificate. The intermediate and server certificates must be sent by the TLS server.
Note:
While possible, there is no relation between siblings, so a chain may branch off.
This is possible but unusual. A CA certificate may sign two or more intermediate CA certificates.
Server certificates signed by the different intermediate CAs will strictly be validated by the hierarchy of the certificate chain.
The other intermediate CA certificate will be unusable for verification of a server certificate.
2. Certificate validity period (whole chain).
The created signature has a validity period, consisting of a validity start date and a validity end date.
Outside of this range, the signature or the certificate with this signature is not valid.
Note that the private key has no validity period and can be used indefinitely in order to create newer valid signatures.
A private key should still be changed at some intervals to minimize the risk of a third party leveraging unauthorized access to the private key: replacing it is a security measure and does not add to the work tasks rather than renewing a server certificate.
As such, replacing the web server's certificate is the standard practice when the certificate validity period has come to an end.
It is best practice to have alert reminders in place for the expiration rather than to be notified of end users facing browser warnings.
Each certificate in the chain has its own validity period, but it must be equal or longer than the certificates it issues.
3. Finally, a certificate contains certificate revocation information.
The process of revocation is necessary if a valid certificate needs to be invalidated. This may be performed if the private key has been compromised by a third party or there was another issue with the use of this certificate.
This is a method for one node to check the validity of the certificate sent by another node.
A received certificate will be checked against its revocation status and, if revoked, rejected. A warning will be issued.
The revocation information is either provided with a Certificate Revocation List (CRL), which is a list of certificate serial numbers issued by this CA, or against an Online Certificate Status Protocol (OCSP) server. This link to the server is stored in the certificate.
Both methods are intended to enable the client to check the certificates' revocation status.
These methods are common among TLS nodes.
The handling of a certificate for the web server will only differ in the UI meant to import the certificate, but devices like FortiGate, FortiAnalyzer, FortiNAC, and FortiWeb all follow the same rules.
One special example is Deep Packet Inspection (DPI). This differs only in one way:
In the illustration above with the signatures, the 'server certificate' is not used. Another certificate with CA properties is the same as rootCA or the subCA.
This scheme is exactly the same, with the exception of the certificate that is the lowest in the chain.
The deep packet inspector needs to be a TLS client to the remote server but also needs to be a TLS server to the actual client.
This is no different from a 'Man in the middle' (MITM) attack.
In a normal case, certificate chain trust will prevent this from happening as the DPI CA certificate must be known to the client.
The public key must be imported on the end station. DPI CAs are typically custom local CAs, not in the end-user key store.
Public CAs that are trusted by everybody do not issue CA certificates.
If DPI is intended for managed clients, the public keys for the certificate(s) can be pushed to the managed clients. Unmanaged clients will have no easy way of receiving public keys.
Note that unmanaged clients, like Guest users usually do not have the public keys/CA certificate of the internal CA. They typically require server certificates, trusted by public certificate authorities. A very common example is a captive portal that is served to external visitors. TLS must be set up correctly for them otherwise it will cause certificate warnings.
Note on file types when handling certificates.
When handling certificates, various file types and contents and 'encodings' can be seen. Very common is the base64 encoded certificate (for Microsoft certificate export 'CER encoded').
Public certificate key material will start with:
'-----BEGIN CERTIFICATE-----' and end with '-----END CERTIFICATE-----'.
Private certificate key material will start with (might differ depending on the key algorithm):
'-----BEGIN RSA PRIVATE KEY-----' and end with '-----END RSA PRIVATE KEY-----'.
The certificates are stored in certain file formats and extensions.
Public keys come usually with a '.cer'/'.crt' extension, sometimes as '.pem'.
Private keys come either as a separate file as '.key' or as a bundle file together with the public key in a PKSC#12 format, file extension '.p12'. They might also be stored as '.pfx'.
In order to use a certificate for identification it is required to have the private key and it is necessary to have either the .key file or a .p12.
A special case is a certificate signing request, that comes with a '.csr'. It is created by a private key on the device that requires one to get a full certificate, for example, a FortiGate can create a certificate signing request. The CSR will have to be signed with a CA's private key, resulting in a public key and a .cer/.crt file. This can ONLY be imported to the device that issued the CSR. In the case of using a CSR, NO private key will be visible or exposed.
An example of the common file types, which can be seen when receiving a certificate from the chosen CA:
- fortigate.crt.
- fortigate.cer
- fortigate.p12.
- fortigate.pem.
- fortigate.key.
- fortigate.pfx.
crt/cer = public key.
p12 = public + private key.
pem = public key(s).
key = private key.
pfx = public + private key.
When importing certificates, make sure to have either:
- A CSR from the requesting device and import the public key back to it again.
- The private + public key material for the device needs to identify itself with certificates.
There are free tools available to work with certificates. Note that these tools are not related to Fortinet, and are generally available and free to use. These are not supported or endorsed by Fortinet.
NMAP: https://nmap.org/.
NMAP is a network scanner and the use of a network scanner may be prohibited in the environment of choice. If unsure, do not run this.
A syntax example of receiving the supported TLS ciphers from a server:
nmap --script ssl-enum-ciphers <address> -p <port>
OpenSSL: https://www.openssl.org/.
A syntax example of using OpenSSL to see a TLS server response and the certificates it sends (not necessarily the complete chain):
openssl s_client -showcerts -connect <address>:<port>
Example output header:
openssl s_client -connect www.fortinet.com:443
CONNECTED(00000003)
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
verify return:1
depth=0 C = US, ST = California, L = Sunnyvale, O = "Fortinet, Inc.", CN = *.fortinet.com
verify return:1
---
Certificate chain
0 s:C = US, ST = California, L = Sunnyvale, O = "Fortinet, Inc.", CN = *.fortinet.com
i:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
1 s:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
2 s:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
---
This is followed by the actual certificate data of the server, and if 'showcerts' was included, the intermediate CA certificates. The chain is displayed here, showing the link from 0 (server certificate) to 2 (root certificate, self-signed). 's' indicates the 'subject', the name of the certificate, valid for the subdomain of 'fortinet.com'. 'i' indicates the certificate that signed this certificate.
Example reading based on the openssl example: 0-subject, issued by 0-issuer. 0-issuer=1-subject, issued by 1-issuer. 1-issuer=2-subject. 2-subject issued by itself = RootCA
This can also be applied to the special TLS case that allows an unencrypted header exchange but allows asking for encryption - STARTTLS. It was made for compatibility when transitioning to SSL/TLS and is still typical for LDAP, SMTP, and FTP.
HTTP does not offer STARTTLS. While STARTTLS is possible, it is a good idea to enforce encryption, rather than making it optional.
openssl s_client -showcerts -starttls smtp -connect <address>:<port>
This will work with TLS1.3 as well as TLS1.2. If the certificates must be visible in a packet capture, then the option -tls1_2 should be used.
When reading certificates and trying to match a certificate to its issuer, two attributes of a certificate can be helpful, if used:
- Subject Key Identifier (SKI).
- Authority Key Identifier (AKI).
The SKI is the identifier of this certificate, the AKI equals the SKI of the issuing certificate. The identifiers look unique but ultimately are not. They can be however a good tool to manually complete a chain or see if an incorrect chain has been sent. This may be more useful when other tools are unavailable, like inside RADIUS EAP. Here, a packet capture may provide the best visibility of the certificates that are exchanged between the nodes.
Related articles about certificates and respective implementations on Fortinet Products:
FortiNAC:
Technical Tip: Concatenate certificate files to create complete certificate bundle files.
Technical Tip: Persistent Agent fails to communicate with 'SSL_get_verify_result' log entry
FortiMail:
Technical Note: Replacing the default FortiMail certificate.
Technical Note: How to import certificate signed by intermediate CA into FortiMail.
FortiGate:
Technical Tip: How to generate a self signed certificate from FortiGate.
Troubleshooting Tip: A guide to FortiGate and certificate issues.
Troubleshooting Tip: RADIUS over TLS in FortiGate (RADSEC)
FortiAnalyzer/FortiManager:
Technical Tip: FortiAnalyzer secure log forwarding
Technical Tip: Configuring LDAPS on FortiManager and FortiAnalyzer
FortiSIEM:
Docs: Configuring FortiSIEM for HTTPS Communication Using Public CA Certificates
FortiAuthenticator is a special case, as it can:
- Act as a CA (and sign/issue certificates for users and services).
- Use CA certificates as TLS clients.
- Act as a TLS server for the web UI, but also as a TLS server for a RADIUS server. Secure TLS sessions are established and encapsulated in RADIUS.
Technical Tip: LDAPS with FortiAuthenticator
See Technical Tip: How to issue EAP certificate with Microsoft certification authority.
FortiTrust ID (FortiAuthenticator Cloud):
Docs: RADSEC SNI – RADIUS clients with FortiTrust Identity (Cloud based 802.1x)
FortiClient EMS:
Troubleshooting Tip: EMS certificate not trusted with customized certificate
