FortiGate
FortiGate Next Generation Firewall utilizes purpose-built security processors and threat intelligence security services from FortiGuard labs to deliver top-rated protection and high performance, including encrypted traffic.
JianWu
Staff
Staff
Article Id 334946
Description This article describes when running into such a message in the system event log, what it means, and provides a guideline for troubleshooting as well as suggestions for log collection to engage Fortinet TAC.
Scope FortiGate Log CMDB deadlock.
Solution

Details on log info can be found at FortiOS Log Message Reference:

44555 - LOGID_EVENT_CMDB_DEADLOCK_DETECTED

 

Here is an example of the log message in the system event log:

logid="0100044555" type="event" subtype="system" level="critical" vd="root" eventtime=1611934024311107220 tz="+0900" logdesc="CMDB lock deadlock is detected." msg="cmdbsvr detected cmdb lock deadlock: lock_pid=190, lock_type=0. Sent SIGSEGV and SIGKILL to pid=190(/bin/newcli)"

 

Another example for the message:

 

cmdbsvr detected cmdb lock deadlock: lock_pid=20984, lock_type=0. Sent SIGSEGV and SIGKILL to pid=20984(/bin/sshd)

 

Meaning of this message mean.

Cmddbsvr --  Command dB server - update processes/configuration.

This log indicated another application forked a process (sshd, or newcli in the example above) that kept the database locked while cmdbsvr daemon tried to update the config (this requires cmdbsvr to have the lock of the config). cmdbsvr would keep trying for 60 seconds, it would then kill the process (ssh/newcli in the above examples, it can be other processes) that locked the database and finished its job.

 

General approach - Troubleshooting.

Usually, the killed process will restart right away, so if it only happened once or rarely with no known traffic interruption, it is less of an issue and may not be a concern.

But if it happens more often or causes traffic interruption, follow the guidelines below and look into the issue.

 

The killed process could be the key info to identify the issue. If the process name is less familiar, refer to the link below:

Technical Tip: How to list processes in FortiOS

Technical Tip: Short list of processes

 

  1. Collect the logs below, this will help to identify the issue needed when discussing with Fortinet TAC.
    1. System event logs before and after (from FortiAnalyzer or FortiGate GUI).
    2. Crash log ('diagnose debug crashlog read').
    3. Run the command 'diagnose sys top 2 50', (make sure the crashed process name shows up).
    4. Depending on the crashed process, the debug output of the process may help 'diagnose debug application xxxx'.
    5. Snapshot of 'get system performance status', more helpful if running during the trouble time.

     

  1. Based on the killed process name and info collected above, check the release note and see if this could be a known issue.

         

Scenarios seen before:

Following are two examples, the case can be different and more complicated than this.

 

logid="0100044555" type="event" subtype="system" level="critical" vd="root" eventtime=1611934024311107220 tz="+0900" logdesc="CMDB lock deadlock is detected." msg="cmdbsvr detected cmdb lock deadlock: lock_pid=190, lock_type=0. Sent SIGSEGV and SIGKILL to pid=190(/bin/newcli)"

 

If having an application running FortiGate config backup on a schedule, try temporarily stop the schedule/process to isolate whether they are related, if so, changing the schedule frequency may help.

 

[cmdbsvr_sigalrm_handler:98] cmdbsvr detected cmdb lock deadlock: lock_pid=1021, lock_type=0. Sent SIGSEGV and SIGKILL to pid=1021(/bin/forticron)

 

FortiCron is responsible for scheduling. Check if auto-script is in use, try temporarily disabling it, or change the schedule and see if it helps.

 

  1. Restart the FortiGate: In some cases, a simple restart of the FortiGate may resolve temporary issues with CMDB locks. This action can help release any stuck resources and restore normal operation. However, this is a temporary solution, and the underlying cause needs to be investigated.