Skip to main content
heljag
Staff
Staff
December 24, 2024

Troubleshooting Tip: Security Fabric automation-stitch not running on dowstream devices

  • December 24, 2024
  • 0 replies
  • 1463 views
Description This article describes how to successfully fix automation-stitches not running on downstream devices.
Scope FortiGate.
Solution diagramKB.png

 

On a Security Fabric topology, the automation-stitch created on the Fabric Root is synchronized and will run on all the downstream devices.

 

There are situations in which that is not happening. Here is an example of an automation-stitch that will add an IP to a block list:

 

config system automation-action
    edit "IPblocked"
        set action-type cli-script
        set script "diagnose user banned-ip add src4 %%log.srcip%% %%log.expiry%% admin"
        set execute-security-fabric enable
        set accprofile "super_admin"
    next
end

 

The option 'execute-security-fabric' is enabled, meaning that this should be executed on the Fabric devices and not only on the Fabric Root. Upon running the stitch, the object was not added to the downstream devices.

 

  • Fabric Root:

 

fabricroot.png

 

  • Downstream device:

 

downstream.png

 

For troubleshooting, the 'diagnose debug app autod -1should be run on both devices.

 

  • Fabric Root:

 

FortiGate-101F # diagnose debug application autod -1
Debug messages will be on for 30 minutes.

FortiGate-101F #
FortiGate-101F # diagnose debug enable
FortiGate-101F # pid:277-__handle_msg()-414: Subscriber:4 received package. pubid:0 pkgid:466 pkg_index:191
pid:277-__handle_pkg_logs()-356: Subscriber:4 processing package size:1102 logs:1 pickup:1
pid:277-__subscr_close_cur_pkg()-140: close package size:1102 logs:1
__action_cli_script_open()-186: cli script action:IPblocked is called. svc ctx:0xba32bb0
accprof:super_admin script:
diagnose user banned-ip add src4 1.1.1.1 999 admin

__cli_script_close()-121: cli script:
autod.1
output:
########## script name: autod.1, offset: 213##########
========= #1, 2024-12-24 02:39:37 ==========
FortiGate-101F diagnose user banned-ip add src4 1.1.1.1 999 admin

======= end of #1, 2024-12-24 02:39:37 ======


__action_cli_script_close()-213: cli script action is done. script:
diagnose user banned-ip add src4 1.1.1.1 30 admin
output:
########## script name: autod.1, offset: 213##########
========= #1, 2024-12-24 02:39:37 ==========
FortiGate-101F diagnose user banned-ip add src4 1.1.1.1 999 admin

======= end of #1, 2024-12-24 02:39:37 ======


pid:277-__handle_msg()-414: Subscriber:4 received package. pubid:0 pkgid:467 pkg_index:190
pid:277-__handle_pkg_logs()-356: Subscriber:4 processing package size:3213 logs:4 pickup:4

 

  • Downstream device:

 

FortiGate-61F # diagnose debug application autod -1
Debug messages will be on for 24 minutes.

FortiGate-61F # diagnose debug enable

FortiGate-61F # pid:231-__handle_msg()-414: Subscriber:4 received package. pubid:0 pkgid:374 pkg_index:191
pid:231-__handle_pkg_logs()-356: Subscriber:4 processing package size:1505 logs:1 pickup:1
pid:231-__subscr_close_cur_pkg()-140: close package size:1505 logs:1
auto_action_relayed()-446: Failed to allocate ctx for relayed stitch.logid:32102 sn:FG101FTK19000431 stitch:IPblocked

 

On the downstream device, the message 'Failed to allocate ctx for relayed' is being thrown. This means that the script is not on the downstream devices.

 

Comparing the security fabric config on the downstream devices, the "configuration-sync" option is set to local.

 

  • Downstream device:

 

config system csf
    set status enable
    set upstream "192.168.250.254"
    set configuration-sync local
end

 

The 'configuration-sync' option was changed from 'local' to 'default' on the downstream device, and the script was run again.

 

  • Downstream device:

 

FortiGate-61F # diagnose debug application autod -1
Debug messages will be on for 30 minutes.

FortiGate-61F # diagnose debug enable

FortiGate-61F # __action_cli_script_open()-186: cli script action:IPblocked is called. svc ctx:0x9dbbc00
accprof: script:
diagnose user banned-ip add src4 1.1.1.1 999 admin

__cli_script_close()-121: cli script:
autod.0
output:
########## script name: autod.0, offset: 214##########
========= #1, 2024-12-24 03:35:43 ==========
FortiGate-61F diagnose user banned-ip add src4 1.1.1.1 999 admin

======= end of #1, 2024-12-24 03:35:43 ======


__action_cli_script_close()-213: cli script action is done. script:
diagnose user banned-ip add src4 1.1.1.1 9999 admin
output:
########## script name: autod.0, offset: 214##########
========= #1, 2024-12-24 03:35:43 ==========
FortiGate-61F diagnose user banned-ip add src4 1.1.1.1 999 admin

======= end of #1, 2024-12-24 03:35:43 ======


pid:231-__handle_msg()-414: Subscriber:4 received package. pubid:0 pkgid:400 pkg_index:0
pid:231-__handle_pkg_logs()-356: Subscriber:4 processing package size:2009 logs:3 pickup:3
pid:231-__subscr_close_cur_pkg()-140: close package size:2009 logs:3

 

The same error is no longer seen in the debug, and if the banned list is checked on the downstream device, the object was created successfully.

 

downstream1.png

 

Notes:

  1. 'banned-ip' is an operational/quarantine list (runtime), not a CMDB object. FortiGate maintains a 'banned IP address list' that can be managed by administrators using:


diagnose user banned-ip


list   List banned IPs.
add    Add banned IP address.
delete Delete banned IP address.
clear  Clear all banned IP addresses.
stat   stat

 

This is an enforcement/runtime list (often described as a quarantine list in documentation and KBs), not a normal firewall object stored as part of the standard policy/object CMDB structure.

FortiOS includes a 'banned-ip-persistency' option to control whether the banned IP list persists across a power cycle.
This setting is about local retention of the banned list on the same FortiGate, not about distributing the list to
other Fabric members.

Configuring the persistency for a banned IP list

 

Security Fabric object synchronization is oriented around 'Global CMDB objects' (Fabric objects) that can be
synchronized across the Fabric when fabric-object unification is enabled. Documentation/CLI references describe
'Global CMDB objects will be synchronized in Security Fabric' (when fabric-object-unification is set to default), and
individual CMDB objects can be marked as fabric-wide global objects via 'fabric-object enable'.

The 'banned-ip' table does not behave like a CMDB object that can be marked as 'fabric-object'; it is a runtime
enforcement list. As a result, when an administrator manually bans an IP on the Fabric Root (GUI/CLI), that update
typically remains local to the root device and is not propagated as a synchronized CMDB object to downstream devices.

 

  1. When the 'IP ban' is executed by an automation stitch configured to run across the Security Fabric, the Fabric Root
    does not 'sync the banned-ip list as configuration'. Instead, the stitch itself is synchronized and executed on each
    downstream device, and each downstream device performs a local 'ban/quarantine' action that updates its own runtime banned list.

     

  2. In this solution, automation stitch + execute-security-fabric works because the 'IP ban' is executed by an automation stitch configured to run across the Security Fabric; the Fabric Root does not 'sync the banned-ip list as configuration'. Instead, the stitch itself is synchronized and executed on each downstream device, and each downstream device performs a local 'ban/quarantine' action that updates its own runtime banned list.

     

  3. Manual banning changes the root’s local runtime table only, and it will not be propagated as a synchronized CMDB object to downstream devices.