Description
Automation stitches can be created to run a CLI script and send an email message when memory exceeds specified thresholds.
This article describes how to create these automation stitches.
Scope
FortiGate.
Solution
To define Memory usage thresholds, run the following configuration:
config system global
set memory-use-threshold-extreme <percent>
set memory-use-threshold-green <percent>
set memory-use-threshold-red <percent>
end
In the above, 'memory-use-threshold-extreme' is used to define the percentage of total RAM used at which memory usage is considered extreme and new sessions are dropped.The default value is 95.
The 'memory-use-threshold-red' threshold is used to define the percentage of total RAM used at which memory usage forces the FortiGate to enter conserve mode. The default value is 88.
Lastly, 'memory-use-threshold-green' defines a percentage value of total RAM used at which memory usage forces the FortiGate to exit conserve mode. The default value is 82.
Configure the automation stitches
To create an automation stitch for high memory usage, follow the steps below.
Create an automation action to run a CLI script:
High memory usage stitch.
To create an automation stitch for high memory usage:
Create an automation action to run a CLI script:
config system automation-action
edit "high_memory_debug"
set action-type cli-script
set script "
diag sys vd list | grep fib
diag sys cmdb info
fnsysctl df -h
fnsysctl ls -la /tmp
fnsysctl ps
diagnose debug crashlog read
get system performance status
get system session status
diagnose sys session full-stat
diagnose sys flash list
diagnose hardware sysinfo conserve
diagnose hardware sysinfo memory
diagnose hardware sysinfo slab
diagnose hardware sysinfo shm
diagnose hardware deviceinfo disk
get system arp
diagnose ip arp list
diagnose ip address list
get router info routing-table all
get router info kernel
diagnose ip rtcache list
dia sys mpstat 1 2
diagnose sys top-mem 20
diagnose sys top 9 99 2"
next
end
Create an automation action to send an email:
config system automation-action
edit "auto_high_memory_email"
set action-type email
set email-to "person@fortinet.com" <----- Replace with email.
set email-subject "CSF stitch alert: high_memory"
set email-body "%%results%%" <----- In Firmware v7.4.0 or higher, the command is set message "%%results%%".
next
end
Create an automation trigger:
config system automation-trigger
edit "auto_high_memory"
set event-type low-memory
next
end
Create an automation stitch:
config system automation-stitch
edit "auto_high_memory"
set trigger "auto_high_memory"
set action "high_memory_debug" "auto_high_memory_email"
next
end
For v7.4.0 or higher, use the following command:
config system automation-stitch
edit "auto_high_memory"
set trigger "auto_high_memory"
config actions
edit 2
set action "high_memory_debug"
set required enable
next
edit 1
set action "auto_high_memory_email"
set required enable
next
end
next
end
Once the memory reaches the threshold, an email will be received.
Example FGT[FGVMEVC------D] Automation Stitch:auto_high_memory is triggered.
########## script name: autod.4 ##########
========== #1, 2020-05-13 07:17:58 ==========
FortiGate-VM64 $ diagnose debug cli 8
Debug messages will be on for 11 minutes.
FortiGate-VM64 $ diagnose debug console timestamp enable
FortiGate-VM64 $ diagnose debug enable
FortiGate-VM64 $ diagnose debug crashlog read
1: 2020-05-02 06:50:56 the killed daemon is /bin/dhcpcd: status=0x0
2: 2020-05-02 06:56:59 the killed daemon is /bin/pyfcgid: status=0x0
3: 2020-05-02 08:55:01 the killed daemon is /bin/ntpd: status=0xf
4: 2020-05-02 09:01:09 the killed daemon is /bin/pyfcgid: status=0x0
5: 2020-05-02 10:01:06 the killed daemon is /bin/pyfcgid: status=0x0
6: 2020-05-02 10:12:28 the killed daemon is /bin/pyfcgid: status=0x0
7: 2020-05-02 10:23:05 the killed daemon is /bin/pyfcgid: status=0x0
8: 2020-05-02 10:26:22 Interface port1 is brought down. process_id=724, process_name="newcli".
Note:
The maximum size of the CLI script action output is 16K characters as of v6.4.0.
