Description | This article describes when running into such a message in the system event log, what it means, and provides a guideline for troubleshooting as well as suggestions for log collection to engage Fortinet TAC. |
Scope | FortiGate Log CMDB deadlock. |
Solution |
Details on log info can be found at FortiOS Log Message Reference: 44555 - LOGID_EVENT_CMDB_DEADLOCK_DETECTED
Here is an example of the log message in the system event log: logid="0100044555" type="event" subtype="system" level="critical" vd="root" eventtime=1611934024311107220 tz="+0900" logdesc="CMDB lock deadlock is detected." msg="cmdbsvr detected cmdb lock deadlock: lock_pid=190, lock_type=0. Sent SIGSEGV and SIGKILL to pid=190(/bin/newcli)"
Another example for the message:
cmdbsvr detected cmdb lock deadlock: lock_pid=20984, lock_type=0. Sent SIGSEGV and SIGKILL to pid=20984(/bin/sshd)
Meaning of this message mean. Cmddbsvr -- Command dB server - update processes/configuration. This log indicated another application forked a process (sshd, or newcli in the example above) that kept the database locked while cmdbsvr daemon tried to update the config (this requires cmdbsvr to have the lock of the config). cmdbsvr would keep trying for 60 seconds, it would then kill the process (ssh/newcli in the above examples, it can be other processes) that locked the database and finished its job.
General approach - Troubleshooting. Usually, the killed process will restart right away, so if it only happened once or rarely with no known traffic interruption, it is less of an issue and may not be a concern. But if it happens more often or causes traffic interruption, follow the guidelines below and look into the issue.
The killed process could be the key info to identify the issue. If the process name is less familiar, refer to the link below: Technical Tip: How to list processes in FortiOS Technical Tip: Short list of processes
Scenarios seen before: Following are two examples, the case can be different and more complicated than this.
logid="0100044555" type="event" subtype="system" level="critical" vd="root" eventtime=1611934024311107220 tz="+0900" logdesc="CMDB lock deadlock is detected." msg="cmdbsvr detected cmdb lock deadlock: lock_pid=190, lock_type=0. Sent SIGSEGV and SIGKILL to pid=190(/bin/newcli)"
If having an application running FortiGate config backup on a schedule, try temporarily stop the schedule/process to isolate whether they are related, if so, changing the schedule frequency may help.
[cmdbsvr_sigalrm_handler:98] cmdbsvr detected cmdb lock deadlock: lock_pid=1021, lock_type=0. Sent SIGSEGV and SIGKILL to pid=1021(/bin/forticron)
FortiCron is responsible for scheduling. Check if auto-script is in use, try temporarily disabling it, or change the schedule and see if it helps.
|