Inbound SMTP STARTTLS deep inspection causes delivery/banner timeout when UTM profiles are enabled

Hi everyone,

I am testing **inbound SMTP STARTTLS deep inspection** on a FortiGate policy that publishes an internal mail gateway behind FortiGate.

I found two Fortinet KB articles that seem directly related to this case: Technical Tip: Inbound email to mail server protected by FortiGate is not logged or inspected by Anti-Spam profile

Technical Tip: Enabling SSL/SSH inspection causes connectivity issues for STARTTLS
 

The first article says that inbound SMTP traffic using SMTPS or STARTTLS, including STARTTLS on port `25`, needs SSL inspection if FortiGate is expected to inspect/log it with Anti-Spam, AV, DLP, etc.

The second article says that SSL/SSH inspection in front of a mail security gateway can cause STARTTLS connectivity issues.

My case looks very similar.

Environment: Internet MTAs → FortiGate VIP → internal SMTP gateway

The internal SMTP gateway is a Postfix / mail security gateway.
The backend SMTP server supports STARTTLS on port `25`.

Public side is published through VIPs:

1.1.1.1:25 → internal-mail-gateway:25
2.2.2.2:25 → internal-mail-gateway:25

The original production policy works fine with `certificate-inspection`:

config firewall policy
    edit 233
        set name "Publish SMTP | WAN → DMZ"
        set srcintf "virtual-wan-link"
        set dstintf "DMZ"
        set action accept
        set srcaddr "all"
        set dstaddr "VIP_SMTP_1" "VIP_SMTP_2"
        set schedule "always"
        set service "SMTP"
        set utm-status enable
        set inspection-mode proxy
        set ssl-ssh-profile "certificate-inspection"
        set av-profile "SMTP"
        set emailfilter-profile "Default Email Filter"
        set ips-sensor "high_security"
        set application-list "EMail Access"
        set profile-protocol-options "default"
        set logtraffic all
        set logtraffic-start enable
    next
end

In this state, mail delivery works normally. MXToolbox also shows normal SMTP behavior:

220 mail.domain.tld Mail Server
250-STARTTLS
SMTP TLS: OK
Open Relay: OK - Not an open relay

The problem is that FortiGate cannot inspect the body/MIME/attachments/URLs inside SMTP traffic after STARTTLS unless deep inspection is enabled. That is why I started testing SMTP STARTTLS deep inspection.

First, I tested the approach recommended in KB 95711:

Protecting SSL Server
Valid public mail server certificate + private key imported into FortiGate
SSL/SSH inspection profile applied to inbound SMTP policy
 

I imported a valid public wildcard certificate for `*.domain.tld` into FortiGate and used it in a `Protecting SSL Server` SSL inspection profile.

Conceptually this seems correct for inbound SMTP.

The certificate/DNS side looked correct. The policy matched and DNAT worked. However, real mail delivery started failing. For example, Gmail connected to FortiGate, policy match and DNAT were visible, but the message did not reach the backend mail gateway. As soon as I rolled back to `certificate-inspection`, delivery worked immediately again.

Then I tested a second approach based on the AV scanning SMTP KB, using CA-based deep inspection with `Fortinet_CA_SSL`.

SSL inspection profile:

config firewall ssl-ssh-profile
    edit "SMTP_STARTTLS_DI"
        set comment "Deep inspection for SMTP STARTTLS on port 25 using FortiGate CA"
        set caname "Fortinet_CA_SSL"

        config https
            set status disable
        end
        config ftps
            set status disable
        end
        config imaps
            set status disable
        end
        config pop3s
            set status disable
        end
        config ssh
            set status disable
        end
        config dot
            set status disable
        end

        config smtps
            set ports 25
            set status deep-inspection
            set client-certificate bypass
            set unsupported-ssl-version allow
            set unsupported-ssl-cipher allow
            set unsupported-ssl-negotiation allow
        end

        set ssl-exemption-ip-rating disable
        set ssl-exemption-log enable
        set ssl-server-cert-log enable
        set ssl-handshake-log enable
    next
end

I also created a custom protocol-options profile for SMTP:

config firewall profile-protocol-options
    edit "SMTP_STARTTLS_PO"
        set comment "Protocol options for SMTP STARTTLS deep inspection"

        config http
            set status disable
        end
        config ftp
            set status disable
        end
        config imap
            set status disable
        end
        config pop3
            set status disable
        end
        config nntp
            set status disable
        end
        config mapi
            set status disable
        end
        config dns
            set status disable
        end
        config cifs
            set status disable
        end

        config smtp
            set ports 25
            set status enable
            set inspect-all disable
            set proxy-after-tcp-handshake enable
            set options fragmail splice
            set oversize-limit 25
            set uncompressed-oversize-limit 25
            set scan-bzip2 enable
            set server-busy disable
            set ssl-offloaded no
        end
    next
end

Important test result:

If I apply only these two profiles:

set ssl-ssh-profile "SMTP_STARTTLS_DI"
set profile-protocol-options "SMTP_STARTTLS_PO"
 

and no UTM profiles, SMTP works. MXToolbox sees the SMTP banner, STARTTLS is advertised, and the transaction is normal.

Example result:

220 mail.domain.tld Mail Server
250-STARTTLS
SMTP TLS: OK
SMTP Transaction Time: OK
 

But as soon as I add any UTM profile, the SMTP banner times out.

For example:

Only AV profile enabled: set av-profile "SMTP"
Result:

Timeout waiting for response after 15 seconds.
Completed Connect
SMTP Banner Check: Reverse DNS does not match SMTP Banner
SMTP TLS: Warning - Does not support TLS
SMTP Transaction Time: 15 seconds
```

Only Application Control enabled: set application-list "EMail Access" - Same result: SMTP banner timeout.

Only Email Filter enabled: set emailfilter-profile "Default Email Filter" Same result: SMTP banner timeout.

The backend SMTP gateway does send the banner:
 

But when UTM profiles are enabled, FortiGate does not appear to release that banner back to the external client. Debug flow shows that the policy is matched, DNAT works, and traffic is sent to IPS / application layer.

After rollback to the original configuration:

set ssl-ssh-profile "certificate-inspection"
set profile-protocol-options "default"
set av-profile "SMTP"
set emailfilter-profile "Default Email Filter"
set ips-sensor "high_security"
set application-list "EMail Access"

everything works again immediately.

My main goal is simple:

Inspect inbound SMTP STARTTLS traffic for AV / Anti-Spam / URL content without requiring external mail servers around the world to trust or install a FortiGate CA certificate.
 

Any guidance from anyone who has successfully deployed inbound SMTP STARTTLS inspection with UTM profiles on FortiGate would be appreciated.