| Automation stitches can be configured to send an alert email when an HA failover event occurs. When multiple FortiGates are in a security fabric, automation stitch is only configurable in the root FortiGate, and will be synchronized to all downstream non-root FortiGates via the security fabric. Refer to the related article for configuration guidance: Technical Tip: How to configure Automation stitch for Downstream FortiGate and confirm it is working. In an example scenario, the downstream device is an HA cluster that requires an automation stitch to send email alerts when an HA failover occurs. Example topology is as follows: Root FortiGate (10.30.2.224) <----- Non-root FortiGate HA cluster (10.30.3.181). As per the article above, the automation stitch for the HA failover is configured in the root FortiGate, and is further synchronized to the downstream FortiGate HA cluster as below: Automation stitch for HA failover event in Root FortiGate: config system automation-stitch edit "HA failover" set trigger "HA Failover" config actions edit 1 set action "Email Notification" set required enable next end next end The following command is executed, and the output confirms that the automation stitch has been synchronized to the downstream HA cluster: diagnose test application autod 2
stitch: HA failover destinations: all trigger: HA Failover type:ha failover
local hit: 0 relayed to: 0 relayed from: 1 actions: Email Notification type:email interval:0 delay:0 required:yes subject: %%log.logdesc%% body: %%log%% sender: mailto:xxxx@gmail.com; When testing the automation stitch on the downstream HA cluster by executing the test command below (without performing an actual HA failover), the alert email can be received: diagnose automation test "HA failover" automation test is done. stitch:HA failover By executing the following debug commands for the automation stitch and security fabric on the primary unit of the HA cluster, it can be validated that the HA cluster informs the root to send the email via the security fabric when the automation stitch is triggered, rather than sending the alert email by itself: diagnose debug application autod -1 diagnose debug application csfd -1 diagnose debug console timestamp enable diagnose debug enable 2026-02-08 21:37:55 <2124-M> 8000 nstd_handle_auto_pkt()-97: received msg (sz:520) from autod. { "msg": 2, "sn": "FGVMxxxxxxxxx", "logid": 37892, "log": "date=2026-02-08 time=21:37:54 devid=\"FGVMxxxxxxxxx\" devname=\"P-kvm69\" eventtime=1770615474232491978 tz=\"-0800\" logid=\"0108037892\" type=\"event\" subtype=\"ha\" level=\"notice\" vd=\"root\" logdesc=\"Virtual cluster member state moved\" ui=\"automation\" action=\"test\" msg=\"automation stitch Test: Virtual cluster's member state moved\" ha_role=\"master\" vcluster=1 vcluster_state=\"work\" vcluster_member=1 hostname=\"test\" sn=\"123456\" stitch=\"HA failover\"", "stitch": { "name": "HA failover", "id": 6 }, "action": { "name": "Email Notification", "id": 0, "last": 1 } } 2026-02-08 21:37:55 <2124-M> 8000 nstd_handle_auto_pkt()-103: packet type:3 2026-02-08 21:37:55 <2124-M> 400000 reliable_messaging_do_send()-1026: 2026-02-08 21:37:55 <2124-M> 400000 reliable_messaging_build_outgoing()-950: 2026-02-08 21:37:55 <2124-M> 200000 reliable_messaging_build_outgoing()-951: Sending message type 3(NSTD_RELIABLE_MSG_AUTOMATION) to FGVMxxxxxxxxx, length 543 2026-02-08 21:37:55 <2124-M> 400000 reliable_set_message_key()-1108: 2026-02-08 21:37:55 <2124-M> 08 nstd_find_next_toward_destination()-426: Next device towards the destination FGVMxxxxxxxxx is FGVMxxxxxxxxx 2026-02-08 21:37:55 <2124-M> 400000 reliable_send_outgoing_message()-1154: 2026-02-08 21:37:55 <2124-M> 10 nstd_send_pkt_fn()-381: Sending packet called from (reliable_send_outgoing_message:1158) 2026-02-08 21:37:55 <2124-M> 10 daemon_send_internal_msg_ex()-211: Sending internal msg NSTD_INTERNAL_MSG_EXTERNAL_PKT data_len=712 The email sent by the root FortiGate can further be validated by executing the following debug commands on the root FortiGate for the 'Alert Email'. The debug output confirms that the root FortiGate successfully processes and sends the email notification. diagnose debug application alertmail -1 Debug messages will be on for 30 minutes. diagnose debug console timestamp enable diagnose debug enable 2026-02-09 17:09:32 Arrived msg(type 9, 515 bytes):xxxx@gmail.com; Virtual cluster member state moved date=2026-02-08 time=21:09:32 devid="FGVMxxxxxxxxx" devname="P-kvm70" eventtime=1770613772854430163 tz="-0800" logid="0108037892" type="event" subtype="ha" level="notice" vd="root" logdesc="Virtual cluster member state moved" ui="automation" action="test" msg="automation stitch Test: Virtual cluster's member state moved" ha_role="master" vcluster=1 vcluster_state="work" vcluster_member=1 hostname="test" sn="123456" stitch="HA failover" 2026-02-09 17:09:32 mail_info: from:fortinet-notifications.com user:DoNotReply@fortinet-notifications.com 2026-02-09 17:09:32 mail_info: reverse path:DoNotReply@fortinet-notifications.com user name:DoNotReply 2026-02-09 17:09:32 to[0]:xxxx@gmail.com 2026-02-09 17:09:32 <==_init_mail_info 2026-02-09 17:09:32 create session 2026-02-09 17:09:32 resolve fortinet-notifications.com to 1 IP 2026-02-09 17:09:32 ==> send mail 2026-02-09 17:09:32 connecting to 208.91.114.151 port 465 2026-02-09 17:09:32 send mail 0x55ffe9058050 session 0x55ffe9039920 2026-02-09 17:09:32 session_io_event: creating ssl structure for session 0x55ffe9039920 2026-02-09 17:09:32 create_ssl: 0x7fd9ab7fc000 2026-02-09 17:09:33 sessionn 0x55ffe9039920, SSL connected 2026-02-09 17:09:34 session: 0x55ffe9039920, rsp_state: greeting, code: 220 2026-02-09 17:09:34 session: 0x55ffe9039920, rsp_state: ehlo, code: 250 2026-02-09 17:09:34 session: 0x55ffe9039920, rsp_state: mail, code: 250 2026-02-09 17:09:34 session: 0x55ffe9039920, rsp_state: rcpt, code: 250 2026-02-09 17:09:35 session: 0x55ffe9039920, rsp_state: data, code: 354 2026-02-09 17:09:35 === send: date=2026-02-08 time=21:09:32 devid="FGVMxxxxxxxxx" devname="P-kvm70" eventtime=1770613772854430163 tz="-0800" logid="0108037892" type="event" subtype="ha" level="notice" vd="root" logdesc="Virtual cluster member state moved" ui="automation" action="test" msg="automation stitch Test: Virtual cluster's member state moved" ha_role="master" vcluster=1 vcluster_state="work" vcluster_member=1 hostname="test" sn="123456" stitch="HA failover" 2026-02-09 17:09:35 session: 0x55ffe9039920, rsp_state: data2, code: 250 2026-02-09 17:09:35 session: 0x55ffe9039920, rsp_state: quit, code: 221 2026-02-09 17:09:35 session finined 2026-02-09 17:09:35 _session_on_destroy 2026-02-09 17:09:35 <== send mail success, m = 0x55ffe9058050 s = 0x55ffe9039920 However, when the real HA failover occurs in the downstream HA cluster, the alert email will not be received. When running debug on the root FortiGate, no Alert Email is generated. Run the following debug commands on both units in the downstream HA cluster to observe the error: diagnose debug application autod -1 diagnose debug application csfd -1 diagnose debug console timestamp enable diagnose debug enable Old primary unit: On the old primary after failover, it is observed that the message from the 'autod' daemon (which controls the automation stitch) is discarded because the unit is no longer the HA master: 2026-02-08 21:18:33 <2124-M> 04 generic_event_ha_sync_plugin()-223: 2026-02-08 21:18:33 <2124-M> 8000 nstd_handle_auto_pkt()-82: Discarding msg from autod - not HA master New primary unit: The new primary receives the HA failover event from autod, attempts to trigger the automation stitch, and tries to send the message upstream: 2026-02-08 21:18:33 <2123-M> 8000 nstd_handle_auto_pkt()-97: received msg (sz:458) from autod. { "msg": 2, "sn": "FGVMxxxxxxxxx", "logid": 37892, "log": "date=2026-02-08 time=21:18:32 devid=\"FGVMxxxxxxxxx\" devname=\"P-kvm70\" eventtime=1770614 312611604933 tz=\"-0800\" logid=\"0108037892\" type=\"event\" subtype=\"ha\" level=\"notice\" vd=\"root\" logdesc=\"Virtual cluster member state moved\" msg=\"Virtual cluster's member state moved\" ha_role=\"primary\" vcluster=1 vcluster_state=\"work\" vcluster_member=0 hostname=\"P-kvm70\" sn=\"FGVMxxxxxxxxx\"", "stit ch": { "name": "HA failover", "id": 2 }, "action": { "name": "Email Notification", "id": 0, "last": 1 } } 2026-02-08 21:18:33 <2123-M> 8000 nstd_handle_auto_pkt()-103: packet type:3 2026-02-08 21:18:33 <2123-M> 400000 reliable_messaging_do_send()-1026: 2026-02-08 21:18:33 <2123-M> 400000 reliable_messaging_build_outgoing()-950: 2026-02-08 21:18:33 <2123-M> 200000 reliable_messaging_build_outgoing()-951: Sending message type 3(NSTD_RELIABLE_MSG_AUTOMATION) to FGVMxxxxxxxxx, length 481 2026-02-08 21:18:33 <2123-M> 02 reliable_messaging_build_outgoing()-961: Tried to send reliable message to self
However, since the new primary requires some time to re-establish security fabric connectivity with the root FortiGate after failover, the security fabric connection is not yet available at that moment. As a result, the message fails to be delivered to the root, and the following error is observed: 2026-02-08 21:18:34 <2123-M> 100 nstd_tree_upd_periodic_timer_hd()-448: 2026-02-08 21:18:34 <2123-M> 100 nstd_tree_send_up_notif()-415: 2026-02-08 21:18:34 <2123-M> 80 nstd_tree_send_up_notif()-419: No valid upstream fgt found.
Therefore, the root FortiGate is not informed to send the email notification, even though the automation stitch is triggered on the new primary in the downstream cluster. This causes the automation stitch for the HA failover event to not work as expected during an actual failover. This issue has been observed in all FortiOS versions in the 7.4 and 7.6 branches and will be resolved in FortiOS 8.0.0, to be released in the middle of April 2026. Workaround: One possible workaround is to add a delay between the trigger and action in the automation stitch as shown below: config system automation-stitch edit "HA failover" set trigger "HA Failover" config actions edit 1 set action "Email Notification" set delay 90 set required enable next end next end The above configuration allows the new primary to wait for 90 seconds before informing the root FortiGate to send the alert email. Once the security fabric connection is re-established within this delay period, the root FortiGate can receive the notification and successfully send the alert email. |