Troubleshooting Tip: How to deal with a Kernel panic

[agrakov]

Description	This article describes how to deal with a Kernel panic.
Scope	FortiGate.
Solution	A Kernel panic is a safety measure taken by the Kernelof the operating system if it cannot handle further operations to prevent further data loss. As a result the system may shut down (freeze) or reboot. To investigate a Kernel panic, kernel debugging needs to be enabled before the issue occurs in order to print verbose Kernel back traces to the console at the time of the Kernel panic. To diagnose the cause of the Kernelpanic, collect the information outlined by this article when the issue occurs and send the debug output to the support team for investigation.  There are 2 methods to collect Kernel debug messages. On FortiGate devices with the comlog feature being present please follow this article: https://community.fortinet.com/t5/FortiGate/Technical-Tip-How-to-use-the-COMLog-feature/ta-p/195390 . It allows for kernel debugging without the need to attach a client to the console port. On devices without this feature present, like all low-end models, Kernel debugging can only be collected via the console port. See the instructions below for this method. Connect a computer to the FortiGate console port for long time monitoring.  When using the Windows application PuTTY, refer to the article: Technical Tip: How to create a log file of a session using PuTTY for more information.   Log all output to a file. With the program PuTTY, change the 'Lines of scrollback' to a high value like for example, 999999999 (Category -> Window -> Lines of scrollback), to view the log file in its entirety. Run the following debug commands. The below example is for a device running in a single VDOM. (Depending on feature availability, some commands may not be accepted. Ignore them.): diagnose debug reset diagnose debug console timestamp enable diagnose debug kernel level 8 diagnose debug duration 0 diagnose debug enable The Kerneldebug level 8 might produce too many log messages. Sometimes it is required to lower the debug level to 4 instead: diagnose debug reset diagnose debug console timestamp enable diagnose debug kernel level 4 diagnose debug duration 0 diagnose debug enable In some cases, it might be required to enable additional debug commands to enable the NMI watchdog, the comlog feature, or debug logging for the IPM sensor daemon: diagnose sys nmi-watchdog enable diagnose debug comlog enable diagnose debug app ipmc -1 In other cases, it might be advised to run the top command to periodically print data to the console to avoid any idle timeouts: diagnose sys top 2 80 Important note: On low-end units (such as FortiWiFi-6xF/4xF models), do not enable any debug commands, connect to the console, and wait for the Kernel panic log to print directly on the console. If Multi-VDOM mode is enabled on FortiGate, navigate to the GLOBAL VDOM by first entering the command config global and then executing the same commands as outlined above: config global Leave the debug commands running until a Kernel Panic occurs. Periodically check the monitoring device to ensure the connection is still alive. When using the PuTTY application, press the 'c' key to refresh the output. The command diag sys top should keep the session active automatically, but will add additional data to the Kernel log lines, which can make it more difficult to read the Kernel debug messages. Use monitoring tools for system outages. Upon finding the failure, mark the date and time. Check the console to see if it is responsive. Press any key to refresh the diag sys top list. If it fails to refresh, try pressing Ctrl + C on the keyboard to stop the command. If the system is still responsive, it will return to the CLI prompt. If the NMI feature was enabled with diagnose sys nmi-watchdog enable, then try holding down the NMI button (applicable only for specific models). Hold the button down for 1 minute. It should output information to the console and reboot the unit automatically. If the system is completely frozen, it may not work. This step is important to try, regardless. The NMI button is typically on the front left of the system: Check the LCD lights and indicate whether the LCD panel is frozen or responsive. After the reboot finishes and the service is restored, complete the following: Gather the kernel debug logs collected from the console client. On devices with the comlog feature, print the outputs of the comlog buffer: diagnose debug comlog read Additionally, it might be helpful to print the outputs of the crash log: diagnose debug crashlog read Alternatively, collect a complete debug report which also contains the crashlog read command: diagnose debug report Gather system event logs. If the system is a cluster, gather the logs from all cluster nodes. Attach all diagnostics to the TAC support case.| After obtaining all necessary information, disable debug logging (optional): diagnose debug reset diagnose debug disable The comlog feature can be cleared and/or disabled. In general, it is advised to keep the comlog feature enabled to capture future kernel panics in case they are observed again. diagnose debug comlog disable diagnose debug comlog clear