Technical Tip: Historical and Realtime Debug Logs for determining RCA
Description
This article describes Historical and Realtime Debug Logs for determining RCA.
Scope
FortiGate.
Solution
Historical Logs:
- Stored data about past system events.
- Covers a wide array of incidents to understand general trends about the system health, network, etc.
- Helps establish a baseline for environmental normalcy and also understand any changes to it over time.
- May not contain sufficient or detailed information for specific incidents, and therefore needs to run real-time debugging.
- Logs fetched from Device Disk/FortiAnalyzer, Syslog, etc: System Event logs, HA Event logs, IPS logs, Traffic Logs, System Debug logs etc.
Realtime Debug Logs:
- Captures live data from a running system, application, or service, and helps quickly understand what is happening in the environment.
Note: For traffic-related issues, it is imperative to gather packet sniffers (not limited to FortiGate) and debug flows to understand where the packet reached/dropped/delayed, etc., before issuing a reboot/failover. Otherwise, it may leave no records to identify the actual underlying issue lying externally or within FortiGate. - Useful for identifying transient or intermittent problems.
- Records data to capture specific issue(s) in detail.
- Logs may sometimes be too verbose, therefore requiring applicable filters to be set before issuance.
- A few debug logs are fetched from the CLI in real-time:
diagnose debug report
diagnose debug application <> -1
diagnose debug enable
diagnose debug kernel level <>
diagnose debug enable
- Serial console logs by issuing NMI (Technical Tip: NMI Button for Troubleshooting Kernel Issues).
- To stop the debug processes in the end defined in the above point, press 'Ctrl+C' and enter 'diagnose debug disable', and to Disable all diagnose debug applications, use the command 'diagnose debug reset'.
Some examples of daemons (applications) are given below;
iked (ike) <-- IPSec VPN.
sslvpnd (sslvpn) <-- SSL VPN.
authd <-- User authentication.
fmbamd <-- Non-blocking authentication daemon.
dhcpd (dhcpc) <-- DHCP client.
dhcps (dhcps) <-- DHCP server.
update <-- FortiGuard communication daemon.
dnsproxy (DNS) <-- DNS daemon.
urlfilter (Web Filter) <-- Web Filter daemon.
Example:
Use IPSec real-time debug:
diagnose debug application ike -1
diagnose debug enable
Enable timestamp:
diagnose debug console timestamp enable
A good use case to run the real-time debug could be used to debug a specific application that was observed crashing at the crash log entry, or an application or daemon consuming high CPU or memory. The 'diagnose sys top' command can be used to find that.
The debug application consumes FortiGate resources, and it is CPU-intensive, even though FortiGate automatically disables debug after 30 minutes. It is recommended to manually disable the debug after it has been used.
For effective RCA, the best approach would be to gather a combination of both real-time debug logs, historical logs for analysis & to purchase the Advanced or Professional Services support contract, which includes official RCA analysis, as RCA analysis is not part of the regular TAC support contract.
Related article:
Technical Tip: How to enhance support experience with TAC engineer
