Fortigate 30E in front of mail server - enabling SSL inspection breaks connections

[train_wreck]

I have a 30E in front of my Arch Linux mail server running Postfix. I have added requisite VIP entries for TCP ports 25,465,587 and 993 port forwarded so the mail server is accessible from the internet. I have downloaded the 30E SSL certificate onto the Linux server and have copied it to the /etc/ssl/certs directory, and Postfix is configured to look in that directory for certs (main.cf has the entry "smtpd_tls_CApath = /etc/ssl/certs").

When I enable SSL inspection for SMTP, I am no longer able to send or receive mail. The only thing I see in the postfix logs is the following:

Mar 30 01:25:10 pLAN9-MX postfix/smtpd[20363]: connect from <OTHERMAILSERVERDOMAIN>[x.x.x.x]

Mar 30 01:25:29 pLAN9-MX postfix/smtpd[20363]: connect from <OTHERMAILSERVERDOMAIN>[x.x.x.x]

Mar 30 01:25:50 pLAN9-MX postfix/smtpd[20363]: connect from <OTHERMAILSERVERDOMAIN>[x.x.x.x]

There is no information as to why the connection is being blocked in the "System Events "log on the Fortigate. If I go to "Security Profiles" -> "Proxy Options" and deselect the "SMTP" option, connections resume, although of course they are not being scanned and so this defeats the purpose of enabling inspection.

Why are the connections being blocked? How do I stop this behavior?

[emnoc]

Q:Did you check the target directory "/etc/ssl/certs"

Q: is the certificate   used by the postfix  includes the CAchain+server.crt ?

Q: did you run diag debug flow on the flow

Q: Was the CA cert added to  the TLS inspection

Q: have you capture what the SSL/TLS server/client hellos and what is provided to postfix

Q: can you run  postfix in debug mode  with the peer list

[train_wreck]

emnoc wrote:

Q: is the certificate   used by the postfix  includes the CAchain+server.crt ?

Your language is confusing; my postfix server has its own certificate generated from my own private CA. This CA has been imported onto the Fortigate. As well, the /etc/ssl/certs directory contains a copy of the Fortigate's SSL inspection certificate. As far as I know, it is a CA certificate itself (it has to be, in order to re-sign the inspected traffic...).

emnoc wrote:
Q: did you run diag debug flow on the flow

Yes, here is the output during a failed connection attempt. Nothing useful I can see here:

id=20085 trace_id=2686 func=resolve_ip6_tuple_fast line=3763 msg="vd-root received a packet(proto=6, 2607:f8b0:4002:c05::22e:35081->2603:3018:1502:62ff::2:25) from wan." id=20085 trace_id=2686 func=resolve_ip6_tuple line=3879 msg="allocate a new session-0001af53" id=20085 trace_id=2686 func=vf_ip6_route_input line=925 msg="find a route: gw-2603:3018:1502:62ff::2 via lan1 err 0 flags 01000001" id=20085 trace_id=2686 func=fw6_forward_handler line=332 msg="Check policy between wan -> lan1" id=20085 trace_id=2686 func=iprope6_fwd_check line=372 msg="in-[wan], out-[lan1], skb_flags-00000000, vid-0, app_id: 0, url_cat_id: 0" id=20085 trace_id=2686 func=__iprope6_tree_check line=561 msg="use addr/intf src tree: len=2 saddr=0:0:0:0:0:0:0:0, eaddr=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff" id=20085 trace_id=2686 func=iprope6_check_one_policy line=988 msg="checked policy-4, ret-matched, act-accept" id=20085 trace_id=2686 func=__iprope6_check line=1115 msg="gnum-4e20, check-7f0ae364" id=20085 trace_id=2686 func=iprope6_check_one_policy line=988 msg="checked policy-4294967295, ret-no-match, act-accept" id=20085 trace_id=2686 func=iprope6_check_one_policy line=988 msg="checked policy-9, ret-no-match, act-accept" id=20085 trace_id=2686 func=iprope6_check_one_policy line=988 msg="checked policy-10, ret-no-match, act-accept" id=20085 trace_id=2686 func=iprope6_check_one_policy line=988 msg="checked policy-11, ret-no-match, act-accept" id=20085 trace_id=2686 func=iprope6_check_one_policy line=988 msg="checked policy-12, ret-matched, act-accept" id=20085 trace_id=2686 func=iprope6_check_one_policy line=1093 msg="policy-12 is matched, act-accept" id=20085 trace_id=2686 func=iprope6_check_one_policy line=1093 msg="policy-4 is matched, act-accept" id=20085 trace_id=2686 func=iprope6_fwd_check line=390 msg="after iprope6_captive_check(): is_captive-0, ret-matched, act-accept, idx-4" id=20085 trace_id=2686 func=fw6_forward_handler line=465 msg="Allowed by Policy-4: AV" id=20085 trace_id=2687 func=resolve_ip6_tuple_fast line=3763 msg="vd-root received a packet(proto=6, 2603:3018:1502:62ff::2:25->2607:f8b0:4002:c05::22e:35081) from local." id=20085 trace_id=2687 func=resolve_ip6_tuple_fast line=3799 msg="Find an existing session, id-0001af53, reply direction" id=20085 trace_id=2688 func=resolve_ip6_tuple_fast line=3763 msg="vd-root received a packet(proto=6, 2607:f8b0:4002:c05::22e:35081->2603:3018:1502:62ff::2:25) from wan." id=20085 trace_id=2688 func=resolve_ip6_tuple_fast line=3799 msg="Find an existing session, id-0001af53, original direction" id=20085 trace_id=2689 func=resolve_ip6_tuple_fast line=3763 msg="vd-root received a packet(proto=6, 2607:f8b0:4002:c05::22e:35081->2603:3018:1502:62ff::2:25) from local." id=20085 trace_id=2689 func=resolve_ip6_tuple_fast line=3799 msg="Find an existing session, id-0001af53, original direction" id=20085 trace_id=2690 func=resolve_ip6_tuple_fast line=3763 msg="vd-root received a packet(proto=6, 2603:3018:1502:62ff::2:25->2607:f8b0:4002:c05::22e:35081) from lan1." id=20085 trace_id=2690 func=resolve_ip6_tuple_fast line=3799 msg="Find an existing session, id-0001af53, reply direction" id=20085 trace_id=2690 func=vf_ip6_route_input line=925 msg="find a route: gw-fe80::7454:7dff:fe80:a7b2 via wan err 0 flags 00450003" id=20085 trace_id=2691 func=resolve_ip6_tuple_fast line=3763 msg="vd-root received a packet(proto=6, 2607:f8b0:4002:c05::22e:35081->2603:3018:1502:62ff::2:25) from local." id=20085 trace_id=2691 func=resolve_ip6_tuple_fast line=3799 msg="Find an existing session, id-0001af53, original direction" id=20085 trace_id=2692 func=resolve_ip6_tuple_fast line=3763 msg="vd-root received a packet(proto=6, 2603:3018:1502:62ff::2:25->2607:f8b0:4002:c05::22e:35081) from lan1." id=20085 trace_id=2692 func=resolve_ip6_tuple_fast line=3799 msg="Find an existing session, id-0001af53, reply direction" id=20085 trace_id=2693 func=resolve_ip6_tuple_fast line=3763 msg="vd-root received a packet(proto=6, 2607:f8b0:4002:c05::22e:35081->2603:3018:1502:62ff::2:25) from local." id=20085 trace_id=2693 func=resolve_ip6_tuple_fast line=3799 msg="Find an existing session, id-0001af53, original direction"

emnoc wrote:
Q: Was the CA cert added to  the TLS inspection

Again, your language is confusing. What does this mean?

emnoc wrote:
Q: have you capture what the SSL/TLS server/client hellos and what is provided to postfix

In regards to my Postfix server, all I see is a TLS helllo on the Postfix server; I see no other traffic beyond that, and Postfix eventually times out the connection due to lack of response and closes the connection.

This is a public mail server; I obviously have no idea what the other end is sending/not sending.

emnoc wrote:
Q: can you run  postfix in debug mode  with the peer list

I am trying this at the moment, will post back if I find anything.

[train_wreck]

So here is slightly more output fro postfix; this is during an attempt to send an email to myself, using my iPhone on Verizon. The iPhone has has the SSL scanning certificate downloaded from the gateway and has been imported correctly into the phone; it shows a green "Verified" in the cert profile configuration screen, and the global trust slider is turned on under "Settings" -> "About phone" -> "Certificate Trust Settings". (Though this should not be necessary to send email since as I mentioned this is a public e-mail server. Of course no one will have my local certificates):

Apr 05 21:08:37 pLAN9-MX postfix/submission/smtpd[552]: warning: database /etc/postfix/aliases.db is older than source file /etc/postfix/aliases Apr 05 21:08:37 pLAN9-MX postfix/submission/smtpd[552]: connect from 57.sub-174-196-159.myvzw.com[174.196.159.57] Apr 05 21:08:37 pLAN9-MX postfix/submission/smtpd[552]: SSL_accept error from 57.sub-174-196-159.myvzw.com[174.196.159.57]: lost connection Apr 05 21:08:37 pLAN9-MX postfix/submission/smtpd[552]: lost connection after STARTTLS from 57.sub-174-196-159.myvzw.com[174.196.159.57] Apr 05 21:08:37 pLAN9-MX postfix/submission/smtpd[552]: disconnect from 57.sub-174-196-159.myvzw.com[174.196.159.57] ehlo=1 starttls=0/1 commands=1/2

So the phone is reporting an "SSL_accept error". Where do I go from here? Will it just not be possible to have the Fortigate scan the traffic for malware/spam if it is SSL encrypted? (If not then it makes it largely useless for what I had in mind).

[kurtli_FTNT]

Hi there, 

   How many firewall policies are using for the mail access? As from your post, the mail server is doing both sending and receiving messages. Also I guess you're using 'deep-inspection' right? Then what the "server-cert-mode" is, "re-sign" or "replace"? Usually the "re-sign " is for outgoing traffic (like PC client accesses internet) and "replace" for incoming. If your postfix is configured well, can you try below for troubleshooting?

1, If single policy used, then split into two policies. One is for outgoing with 're-sign', the other for incoming with 'replace', half-mode is preferred. The idea behind is that you will know in which direction the inspection doesn't work, sending msg or receiveing msg.

2, Try to use the default certificate-inspection other than deep-inspection, see if it works. The certificate-inspection checks less.

3, Is this issue only happening with smtp? Do you have the same issue with a web server?

Thanks.

[train_wreck]

kurtli_FTNT wrote:

Hi there, 

   How many firewall policies are using for the mail access? As from your post, the mail server is doing both sending and receiving messages. Also I guess you're using 'deep-inspection' right? Then what the "server-cert-mode" is, "re-sign" or "replace"? Usually the "re-sign " is for outgoing traffic (like PC client accesses internet) and "replace" for incoming. If your postfix is configured well, can you try below for troubleshooting?

1, If single policy used, then split into two policies. One is for outgoing with 're-sign', the other for incoming with 'replace', half-mode is preferred. The idea behind is that you will know in which direction the inspection doesn't work, sending msg or receiveing msg.

I have 2 IPv4 policies for the mail server; one is from interface form which the mail server is connected to the WAN port; this has NAT enabled as well as SSL inspection, proxy, Antivirus, and Antispam. It is doing deep inspection using a custom policy (since the Fortigate Cookbook said this is necessary).

The second policy is a port forward inbound from the WAN to the MX interface for TCP port 25,465,587 and 993. This has the same settings (except without NAT). If I enable SSL inspection here, this is when incoming mail breaks.

I tried changing the policy from "re-sign" to "replace" on the second, and then the first policy. Nothing changed. Postfix still throws an SSL_accept error.

kurtli_FTNT wrote:

2, Try to use the default certificate-inspection other than deep-inspection, see if it works. The certificate-inspection checks less.

Sorry, but that largely defeats the purpose of having an SSL scanning firewall in the first place. I need to decrypt the encrypted mail to scan it for spam purposes. Without deep inspection, the device is almost useless for antispam, since (according to Google's statistics) over 75% of mail traffic today is encrypted.

kurtli_FTNT wrote:

3, Is this issue only happening with smtp? Do you have the same issue with a web server?

I don't have a web server. I am only running an SMTP/IMAPS server. Curiously, IMAP has no issue (I am using courier as the IMAP server), it never has any problems letting users send and receive mail. It is only when other mail servers try to send mail to me that I have this problem.

Thanks for the help; any other ideas?

[kurtli_FTNT]

So the first policy is for outgoing and second is for incoming. I think there are a few variables in your env and setting, so better isolate them one by one. For example, for troubleshooting purpose only, change ssl-deep-inspeciton on the first policy to 'certificate-inspection'. With certificate-inspection, FGT doesn't do re-sign, but checking the validation and SNI of certificate. Most likely, sending msg should be working with that. If so, dig more, e.g. what the CA certs for ssl-deep-inspection that you put into  "/etc/ssl/certs" are, "Fortinet_CA_SSL" or "Fortinet_CA_Untrusted", or both? And have you tried to use "$smtpd_tls_CAfile", they're similar but this one looks more specific. Also What does the capture on mail server tell us? When and where did Postfix throw error,  in client hello, server hello, or others?

For the incoming messages, you can try to use virtual server with ssl half-mode rather than a static VIP that you have. With that, the diagram looks like:  

Internet===<smtps>===(465):virtual server on FGT---<smtp>---(25):postfix.

The right side traffic is plaintext, and connected to postfix:25, not 465. Below is an example.

===

config firewall vip edit "smtps" set type server-load-balance set extip 22.22.22.140    <=== your VIP set extintf "port22" set server-type smtps set ldb-method round-robin

set extport 465 config realservers edit 1 set ip 11.11.11.140     <=== your postfix set port 25 next end set ssl-certificate "Fortinet_SSL"     <=== use your own certificate here

set ssl-mode half

end

Add this virtual-server into policy

config firewall policy edit 2 set srcintf "port22" set dstintf "any" set srcaddr "all" set dstaddr "smtps" set action accept set schedule "always" set service "ALL" set utm-status enable set av-profile "default" set ips-sensor "default" set profile-protocol-options "default" set ssl-ssh-profile "deep-inspection" set nat enable next end

With above configs, you can see and check the content of incoming messages. Note that since you have several services running on the same VIP, you probably need more VIPs to do the test without interrupting other traffic like IMAP. Also, in some FOS versions, you have to set inspection-mode to proxy under "config system settings" to get ssl-mode and ssl-certficate show up.

Hope that helps.