Description
This article describes the use of the performance statistics system event logs and how to adjust their frequency or disable it when needed.
Solution
FortiOS has a feature that creates a periodic log entry in the system event logs with general information about the performance of the unit.
It includes information about the average CPU load and memory usage, together with the current bandwidth, concurrent total session count and setup rate. See the sample below:
itime=2020-09-01 11:13:06 vd=root bandwidth=13/16 totalsession=53 disk=1 dstepid=3 devid=FGVM04TM20001209 disklograte=0 logdesc=System performance statistics msg=Performance statistics: average CPU: 0, memory: 48, concurrent sessions: 53, setup-rate: 0 idseq=267453240667275345 type=event eventtime=1598951586 mem=48 dtime=2020-09-01 11:13:06 devname=FGT-VM-HEL setuprate=0 dsteuid=0 itime_t=1598951586 euid=3 fazlograte=20 date=2020-09-01 level=notice epid=3 logid=0100040704 subtype=system time=11:13:06 action=perf-stats cpu=0
Features:
-
What it monitors:
- CPU Usage: Understand how much of the average CPU resource is being utilized.
- Memory Usage: Gauge memory consumption over time.
- Concurrent Sessions: Provides insights into the number of simultaneous sessions the unit is handling.
- Setup-Rate: Measures the rate at which new sessions are being set up.
-
Value in Troubleshooting:
- Being able to view historical data can help in correlating certain performance issues to specific events or changes.
- Gives a granular understanding of when and how resource consumption peaks or drops.
This regular report can be very useful when troubleshooting changes in the behavior of the units in terms of resource usage because it provides records to track those changes over time.
However, in some situations, this feature is unwanted because of the frequency of these reports, so there is a configuration parameter to be modified in those cases.
The option is available just from the CLI, and it is as follows:
config system global
set sys-perf-log-interval <0-15>
end
Where:
0 <----- Disabled.
1-15 <----- Time in minutes between logging. 5 is the default value.
To change the CPU use threshold, the following command can be used:
config system global
set cpu-use-threshold "Enter an integer value from <50> to <99>"
end
By default, the CPU usage is logged at the configured 'sys-perf-log-interval' every 5 minutes.
Additionally, if the configured threshold 'cpu-use-threshold' is reached.
Note:
The CPU usage is an average across all CPU cores.
If the device has many cores and a single core spikes, then the average usage of all cores will increase only slightly.
To see a single CPU core spiking in more detail, the feature 'log-single-cpu-high' should be used.
Refer to the article Technical Tip: Enable logging for single core CPU spike against System Event logs.
Best Practices:
-
Monitoring Frequency: While it is tempting to log everything, consider the impact on system performance and storage. Adjust the frequency based on the criticality of monitoring needs.
-
External Storage: Leveraging solutions like FortiAnalyzer can provide a comprehensive view over a more extended period, helpful in trend analysis.
-
Alerts and Notifications: Consider setting up alerts for abnormal behavior. Sudden spikes in CPU or memory usage might be indicative of potential issues.
-
Data Correlation: When analyzing the logs, try to correlate data spikes or abnormalities with other events or changes in the network. This holistic view can help pinpoint issues faster.
Note:
Performance statistics are not logged to disk. It is showing in memory. Performance statistics can be received by a syslog server or by FortiAnalyzer.
For FortiGates with VDOM enabled, the performance stats are logged in the root VDOM only.
The Performance Statistics Logs are a crucial tool in the arsenal of FortiGate administrators, allowing for proactive monitoring and faster troubleshooting. Properly configured, it will provide invaluable insights without overwhelming system resources.
Related documents:
Log and Report
Technical Note: No system performance statistics logs
