Skip to main content
GeorgeZhong
Staff & Editor
Staff & Editor
February 24, 2026

Troubleshooting Tip: Automation stitch for HA failover on downstream FortiGate HA cluster does not send email when triggered by real failover

  • February 24, 2026
  • 1 reply
  • 343 views
Description This article describes a scenario where an automation stitch configured for HA failover is successfully synchronized from the root FortiGate to a downstream HA cluster in the security fabric, but the alert email is not sent when a real HA failover occurs.
Scope FortiGate.
Solution

Automation stitches can be configured to send an alert email when an HA failover event occurs.

 

When multiple FortiGates are in a security fabric, automation stitch is only configurable in the root FortiGate, and will be synchronized to all downstream non-root FortiGates via the security fabric. Refer to the related article for configuration guidance: Technical Tip: How to configure Automation stitch for Downstream FortiGate and confirm it is working.

 

In an example scenario, the downstream device is an HA cluster that requires an automation stitch to send email alerts when an HA failover occurs. Example topology is as follows:

 

Root FortiGate (10.30.2.224) <----- Non-root FortiGate HA cluster (10.30.3.181).

 

As per the article above, the automation stitch for the HA failover is configured in the root FortiGate, and is further synchronized to the downstream FortiGate HA cluster as below:

 

Automation stitch for HA failover event in Root FortiGate:

 

config system automation-stitch
    edit "HA failover"
        set trigger "HA Failover"
            config actions
                edit 1
                    set action "Email Notification"
                    set required enable
                next
            end
    next
end

 

The following command is executed, and the output confirms that the automation stitch has been synchronized to the downstream HA cluster:

 

diagnose test application autod 2

stitch: HA failover
        destinations: all
        trigger: HA Failover
                type:ha failover

        local hit: 0 relayed to: 0 relayed from: 1
        actions:
                Email Notification type:email interval:0
                        delay:0 required:yes
                        subject: %%log.logdesc%%
                        body: %%log%%
                        sender:

 


                        mailto:xxxx@gmail.com;

 

When testing the automation stitch on the downstream HA cluster by executing the test command below (without performing an actual HA failover), the alert email can be received:

 

diagnose automation test "HA failover"
automation test is done. stitch:HA failover

 

By executing the following debug commands for the automation stitch and security fabric on the primary unit of the HA cluster, it can be validated that the HA cluster informs the root to send the email via the security fabric when the automation stitch is triggered, rather than sending the alert email by itself:

 

diagnose debug application autod -1
diagnose debug application csfd -1
diagnose debug console timestamp enable
diagnose debug enable

 

2026-02-08 21:37:55 <2124-M> 8000 nstd_handle_auto_pkt()-97: received msg (sz:520) from autod.
{ "msg": 2, "sn": "FGVMxxxxxxxxx", "logid": 37892, "log": "date=2026-02-08 time=21:37:54 devid=\"FGVMxxxxxxxxx\" devname=\"P-kvm69\" eventtime=1770615474232491978 tz=\"-0800\" logid=\"0108037892\" type=\"event\" subtype=\"ha\" level=\"notice\" vd=\"root\" logdesc=\"Virtual cluster member state moved\" ui=\"automation\" action=\"test\" msg=\"automation stitch Test: Virtual cluster's member state moved\" ha_role=\"master\" vcluster=1 vcluster_state=\"work\" vcluster_member=1 hostname=\"test\" sn=\"123456\" stitch=\"HA failover\"", "stitch": { "name": "HA failover", "id": 6 }, "action": { "name": "Email Notification", "id": 0, "last": 1 } }
2026-02-08 21:37:55 <2124-M> 8000 nstd_handle_auto_pkt()-103: packet type:3
2026-02-08 21:37:55 <2124-M> 400000 reliable_messaging_do_send()-1026:
2026-02-08 21:37:55 <2124-M> 400000 reliable_messaging_build_outgoing()-950:
2026-02-08 21:37:55 <2124-M> 200000 reliable_messaging_build_outgoing()-951: Sending message type 3(NSTD_RELIABLE_MSG_AUTOMATION) to FGVMxxxxxxxxx, length 543
2026-02-08 21:37:55 <2124-M> 400000 reliable_set_message_key()-1108:
2026-02-08 21:37:55 <2124-M> 08 nstd_find_next_toward_destination()-426: Next device towards the destination FGVMxxxxxxxxx is FGVMxxxxxxxxx
2026-02-08 21:37:55 <2124-M> 400000 reliable_send_outgoing_message()-1154:
2026-02-08 21:37:55 <2124-M> 10 nstd_send_pkt_fn()-381: Sending packet called from (reliable_send_outgoing_message:1158)
2026-02-08 21:37:55 <2124-M> 10 daemon_send_internal_msg_ex()-211: Sending internal msg NSTD_INTERNAL_MSG_EXTERNAL_PKT data_len=712

 

The email sent by the root FortiGate can further be validated by executing the following debug commands on the root FortiGate for the 'Alert Email'. The debug output confirms that the root FortiGate successfully processes and sends the email notification.

 

diagnose debug application alertmail -1
Debug messages will be on for 30 minutes.

diagnose debug console timestamp enable

diagnose debug enable

2026-02-09 17:09:32 Arrived msg(type 9, 515 bytes):xxxx@gmail.com;

Virtual cluster member state moved
date=2026-02-08 time=21:09:32 devid="FGVMxxxxxxxxx" devname="P-kvm70" eventtime=1770613772854430163 tz="-0800" logid="0108037892" type="event" subtype="ha" level="notice" vd="root" logdesc="Virtual cluster member state moved" ui="automation" action="test" msg="automation stitch Test: Virtual cluster's member state moved" ha_role="master" vcluster=1 vcluster_state="work" vcluster_member=1 hostname="test" sn="123456" stitch="HA failover"

2026-02-09 17:09:32 mail_info:
from:fortinet-notifications.com user:DoNotReply@fortinet-notifications.com
2026-02-09 17:09:32 mail_info:
reverse path:DoNotReply@fortinet-notifications.com
user name:DoNotReply
2026-02-09 17:09:32 to[0]:xxxx@gmail.com
2026-02-09 17:09:32 <==_init_mail_info
2026-02-09 17:09:32 create session
2026-02-09 17:09:32 resolve fortinet-notifications.com to 1 IP
2026-02-09 17:09:32 ==> send mail
2026-02-09 17:09:32 connecting to 208.91.114.151 port 465
2026-02-09 17:09:32 send mail 0x55ffe9058050 session 0x55ffe9039920
2026-02-09 17:09:32 session_io_event: creating ssl structure for session 0x55ffe9039920
2026-02-09 17:09:32 create_ssl: 0x7fd9ab7fc000
2026-02-09 17:09:33 sessionn 0x55ffe9039920, SSL connected
2026-02-09 17:09:34 session: 0x55ffe9039920, rsp_state: greeting, code: 220
2026-02-09 17:09:34 session: 0x55ffe9039920, rsp_state: ehlo, code: 250
2026-02-09 17:09:34 session: 0x55ffe9039920, rsp_state: mail, code: 250
2026-02-09 17:09:34 session: 0x55ffe9039920, rsp_state: rcpt, code: 250
2026-02-09 17:09:35 session: 0x55ffe9039920, rsp_state: data, code: 354
2026-02-09 17:09:35 === send: date=2026-02-08 time=21:09:32 devid="FGVMxxxxxxxxx" devname="P-kvm70" eventtime=1770613772854430163 tz="-0800" logid="0108037892" type="event" subtype="ha" level="notice" vd="root" logdesc="Virtual cluster member state moved" ui="automation" action="test" msg="automation stitch Test: Virtual cluster's member state moved" ha_role="master" vcluster=1 vcluster_state="work" vcluster_member=1 hostname="test" sn="123456" stitch="HA failover"

2026-02-09 17:09:35 session: 0x55ffe9039920, rsp_state: data2, code: 250
2026-02-09 17:09:35 session: 0x55ffe9039920, rsp_state: quit, code: 221
2026-02-09 17:09:35 session finined
2026-02-09 17:09:35 _session_on_destroy
2026-02-09 17:09:35 <== send mail success, m = 0x55ffe9058050 s = 0x55ffe9039920

 

However, when the real HA failover occurs in the downstream HA cluster, the alert email will not be received. 

 

When running debug on the root FortiGate, no Alert Email is generated.

 

Run the following debug commands on both units in the downstream HA cluster to observe the error:

 

diagnose debug application autod -1
diagnose debug application csfd -1
diagnose debug console timestamp enable
diagnose debug enable

 

Old primary unit:

 

On the old primary after failover, it is observed that the message from the 'autod' daemon (which controls the automation stitch) is discarded because the unit is no longer the HA master:

 

2026-02-08 21:18:33 <2124-M> 04 generic_event_ha_sync_plugin()-223:
2026-02-08 21:18:33 <2124-M> 8000 nstd_handle_auto_pkt()-82: Discarding msg from autod - not HA master

 

New primary unit:

 

The new primary receives the HA failover event from autod, attempts to trigger the automation stitch, and tries to send the message upstream:

 

2026-02-08 21:18:33 <2123-M> 8000 nstd_handle_auto_pkt()-97: received msg (sz:458) from autod.
{ "msg": 2, "sn": "FGVMxxxxxxxxx", "logid": 37892, "log": "date=2026-02-08 time=21:18:32 devid=\"FGVMxxxxxxxxx\" devname=\"P-kvm70\" eventtime=1770614
312611604933 tz=\"-0800\" logid=\"0108037892\" type=\"event\" subtype=\"ha\" level=\"notice\" vd=\"root\" logdesc=\"Virtual cluster member state moved\" msg=\"Virtual
cluster's member state moved\" ha_role=\"primary\" vcluster=1 vcluster_state=\"work\" vcluster_member=0 hostname=\"P-kvm70\" sn=\"FGVMxxxxxxxxx\"", "stit
ch": { "name": "HA failover", "id": 2 }, "action": { "name": "Email Notification", "id": 0, "last": 1 } }
2026-02-08 21:18:33 <2123-M> 8000 nstd_handle_auto_pkt()-103: packet type:3
2026-02-08 21:18:33 <2123-M> 400000 reliable_messaging_do_send()-1026:
2026-02-08 21:18:33 <2123-M> 400000 reliable_messaging_build_outgoing()-950:
2026-02-08 21:18:33 <2123-M> 200000 reliable_messaging_build_outgoing()-951: Sending message type 3(NSTD_RELIABLE_MSG_AUTOMATION) to FGVMxxxxxxxxx, length 481
2026-02-08 21:18:33 <2123-M> 02 reliable_messaging_build_outgoing()-961: Tried to send reliable message to self

However, since the new primary requires some time to re-establish security fabric connectivity with the root FortiGate after failover, the security fabric connection is not yet available at that moment. As a result, the message fails to be delivered to the root, and the following error is observed:

 

2026-02-08 21:18:34 <2123-M> 100 nstd_tree_upd_periodic_timer_hd()-448:
2026-02-08 21:18:34 <2123-M> 100 nstd_tree_send_up_notif()-415:
2026-02-08 21:18:34 <2123-M> 80 nstd_tree_send_up_notif()-419: No valid upstream fgt found.

Therefore, the root FortiGate is not informed to send the email notification, even though the automation stitch is triggered on the new primary in the downstream cluster.

 

This causes the automation stitch for the HA failover event to not work as expected during an actual failover. This issue has been observed in all FortiOS versions in the 7.4 and 7.6 branches and will be resolved in FortiOS 8.0.0, to be released in the middle of April 2026.

 

Workaround: 

One possible workaround is to add a delay between the trigger and action in the automation stitch as shown below:

 

config system automation-stitch
    edit "HA failover"
        set trigger "HA Failover"
            config actions
                edit 1
                    set action "Email Notification"

                    set delay 90
                    set required enable
                next
            end
    next
end

 

The above configuration allows the new primary to wait for 90 seconds before informing the root FortiGate to send the alert email. Once the security fabric connection is re-established within this delay period, the root FortiGate can receive the notification and successfully send the alert email.

    1 reply

    pjang
    Staff & Editor
    Staff & Editor
    March 13, 2026

    Excellently written article, very thorough!