Troubleshooting Tip: FortiWeb coredump HA Failover

[shafiq23]

Description	This article describes how to enable coredump HA failover and demonstrates its usage.
Scope	FortiWeb, FortiWeb-VM HA.
Solution	A coredump file is generated if the system encounters a crash or the daemon restarts due to unexpected circumstances. Coredump file is mainly used by TAC to locate errors in system/daemons that could be used as a temporary workaround and fixed in future firmware releases. Coredump file could get truncated or damaged (mainly proxyd) whereby hardware or virtual platform with higher memory capacity in some cases requires more time to fully generate coredump file. FortiWeb generally takes a maximum of 2 minutes to generate coredump file before restarting the process. Sample of FortiWeb’s kernel log that shows coredump file being truncated: Nov 27 10:12:23 FortiWeb-3000E user.info kernel: [7821159.432316] proxyd[8520]: segfault at 262d1bd6df ip 00007f2815467dc9 sp 00007f27297f2fa0 error 4 Nov 27 10:12:23 FortiWeb-3000E user.info kernel: [7821159.432322] Code: 48 8b 71 08 4c 8b 71 10 48 83 c0 01 48 89 44 24 20 eb 04 48 83 c3 01 49 83 ef 07 49 b8 88 84 99 32 28 7f 00 00 49 3b df 73 25 <48> 0f b6 43 07 49 0f b6 04 00 48 03 d8 48 83 f8 00 75 e8 48 0f b6 Nov 27 10:14:23 FortiWeb-3000E user.warn kernel: [7821279.445442] The core-dumping for proxyd:8520 isn't finished in 120 seconds, truncate it! Substantially, this would cause web application traffic interrupted because of buffer time taken by FortiWeb to generate coredump files. Since v7.0.4GA, a CLI command was introduced to allow FortiWeb to trigger HA failover when proxyd coredump is being generated. config server-policy settings      set corefile-ha-failover enable end This would minimize traffic impact when the secondary unit immediately takes over the traffic thus allowing the primary unit to completely generate coredump files to provide more information for troubleshooting. Note: Refer to the FortiWeb administration guide for more information on corefile-ha-failover What to do when coredump files are truncated or damaged Under Log&Report -> Log Access -> Event, observe the corefile HA monitor process taking place. With HA override enabled: CLI command corefile-ha-failover is enabled. Coredump file generation triggered corefile HA monitor. HA failover takes place.  The coredump file was generated completely and corefile HA monitor stopped. Failback to the previous primary unit due to Priority monitor. With HA override disabled: The slave unit will assume the Primary role after corefile HA monitor starts. No automatic priority monitor failback Export the coredump file: Collecting core/coredump files and logs