|
Challenge Noticed:
CPU spikes may occur randomly, posing a challenge in identifying the root cause.
Observation:
One method to detect a CPU spike is through FortiSwitch logs.
FortiSwitch CLI Command:
execute log display
2022-10-06 11:52:49 log_id=0103035242 type=event subtype=system pri=warning vd=root user="alertd" msg="[First Event] CPU_SENSOR (76.0%) cleared warning threshold of (85.0%)."
As seen above, multiple such events can be reported in the log display output. The event log ID in this case is 0103035242.
Using this log ID create an automation stitch on FortiSwitch to determine which process caused the spike.
Troubleshooting:
To understand what process caused the spike, a dedicated PC should be connected to the FortiSwitch via SSH or console (serial).
Important Note:
The PC should remain connected to the FortiSwitch throughout the monitoring process.
Open a CLI session to FortiSwitch and log the session using the following command. (Use a terminal application that supports logging, such as PuTTY or TeraTerm).
FortiSwitch Automation Setup:
config system automation-trigger
edit "alertd"
set event-type event-log
set logid 0103035242
next
end
config system automation-action
edit "alertd"
set action-type cli-script
set script "fn top -bn 25"
set accprofile "prof_admin"
next
end
config system automation-stitch
edit "alertd"
set trigger "alertd"
set action "alertd"
next
end
Description:
Specify the event trigger by log ID: Under config system automation-trigger, create a trigger named 'alertd', set the event type to event-log, and specify the log ID 0103035242.
Specify the action to run a command: Under config system automation-action, create an action named 'alertd' to run the CLI command "fn top -bn 25", which will be executed 25 times when the trigger occurs.
Create the automation stitch: Under config system automation-stitch, link the trigger and action.
Run the following commands on FortiSwitch to ensure proper monitoring:
config system global
set admintimeout 480
end
diagnose debug console timestamp enable
diagnose debug application autod 8
diag debug enable
Note:
The admintimeout is set to 480 (i.e., 8 hours) so the CLI session (SSH/console) will remain active for that duration. This controls the amount of inactive time before the administrator must log in again. Ensure to re-initiate the connection or press 'Enter' before the timeout.
Monitoring:
Wait for the CPU spike to occur. Once it does, stop the debug using the following command:
diag debug reset
Then, remove the automation stitch and trigger:
config system automation-stitch
delete "alertd"
next
end
config system automation-trigger
delete "alertd"
next
end
config system automation-action
delete "alertd"
next
end
Additional Logs for TAC:
If need to report the issue to TAC, in addition to the automation stitch logs, collect the following logs from the FortiSwitch:
show full-configuration
diagnose debug crashlog read
diag debug report
|
