Description
This article describes how to collect logs for troubleshooting purposes after a PSU failure occurs on a FortiGate 7000E series chassis.
Scope
FortiGate 7000E series Shelf Manager Module (SMM).
Solution
This article contains useful commands that will help in troubleshooting a PSU failure in a FortiGate 7000E series chassis. To run the command below, physical access to a FortiGate 7000E unit (or a remote console port connection to it) is required.
FortiGate 7030E or 7040E units have 4 available PSU slots and are set up by default with 2+1 PSUs configuration. Two PSUs provide the sufficient power and the third PSU is used for redundancy. A forth PSU can be also added to have further power redundancy.
FortiGate 7060E units have 6 available PSU slots and are setup by default with 3+1 PSUs configuration, so three PSUs provide the sufficient power and the forth PSU is used for redundancy. Two more PSUs (fifth and sixth) can be also added to have further power redundancy.
If one PSU fails, the unit will still have enough power to run the FIMs and FPMs modules.
If two PSUs fail, the unit will not have enough power to keep all the slots up and running, which will cause one or more slots will be shutdown (usually FPMs are shutdown first).
For details on how to connect to the SMM module, read the following article: https://community.fortinet.com/t5/FortiGate/Technical-Tip-How-to-access-the-console-port-on-Manageme...
Note: make sure to manually type/spell the full command correctly. Tab autocomplete is not available on the SMM CLI.
- The 'help' command will print out the available SMM command list.
- The 'status' command will print out the power status of the current SMM module. (FGT-7060E units have SMM1 and SMM2 and only one can be active. On the passive SMM, the status command will display the message 'SMM is inactive').
The 'sensor' command will print out the current analog reading of each sensor. Some of the sensors can also display a description (PSU status can show as one of the following: Present, Absent, Input-Lost).
- The 'time get' command will display the current SMM time and date.
- The 'time set <yyyy/mm/dd hh:mm:ss>' command will set the SMM time and date.
- The 'time sync' command will syncronise each slot with the current SMM time and date (FortiOS time is managed by the NTP config in FortiOS and can be different from the SMM time).
The 'sensor_thresholds' command will display the lower (L) and upper (U) thresholds of each sensor. The screenshot below is taken from a FortiGate 7040E unit with PSU1, PSU2 and PSU4 installed, and with an absent PSU3.
- Lower Non-Recoverable (LNR)
- Lower Critical Recoverable (LCR)
- Lower Non-Critical (LNC)
- Upper Non-Critical (UNC)
- Upper Critical Recoverable (UCR)
- Upper Non-Recoverable (UNR)
- The 'sel' command will display the SMM system event logs. Information such as which PSU went down or which FPM or FIM lost power can be found here, along with related timestamps (assuming that the SMM time/date is correctly set). The screenshot below is taken from a FGT-7060E with 4 PSUs (3+1 configuration). Note that the sel command output can be very long. Do not interrupt it, the most recent logs are shown at the end.
Note:
The FortiGate7060E has two SMM modules (SMM1 and SMM2) and only one can be active at any time. The other one acts as standby SMM. If the related sel logs cannot be found in one SMM, a SMM switch-over using the command smm_switch can be forced to make the other SMM active and collect the sel logs from the other SMM module. Once switched over you need to move the cable to the other SMM to access its SMM console (physical access to the chassis unit will be necessary).
Useful links:
1) FortiGate 7000E series datasheet: https://www.fortinet.com/content/dam/fortinet/assets/data-sheets/FortiGate_7000_Series_Bundle.pdf
