Troubleshooting Tip: Investigating unexpected FortiGate reboot events triggered by IPMC Event logs

Description

This article describes a troubleshooting methodology to investigate unexpected FortiGate reboot events associated with IPMC Event logs and PCI heartbeat messages.

Scope

FortiGate, FortiOS v7.x and later.

Solution

Overview:

Unexpected reboot events associated with IPMC Event logs may indicate hardware monitoring actions triggered by the Intelligent Platform Management Controller (IPMC).

The following log messages may be observed:

Emergency reboot, pending action is to reboot in 308 seconds, pci heartbeat[2,5].
User shutdown the device from ipmc_sensord. The reason is 'Shutdown before reboot device.

These events indicate that the IPMC subsystem detected a hardware communication issue and initiated a controlled reboot sequence.

Observed behavior:

The following events were identified:

1. IPMC Event logs with Emergency severity.

2. Multiple interface link-down events before reboot.

3. Shutdown initiated by 'ipmc_sensord' process.

4. Last reboot reason reported as 'power cycle'.

Technical explanation:

The IPMC subsystem continuously monitors hardware health and communication between internal hardware components.

The PCI heartbeat message indicates that heartbeat communication between monitored PCI devices was interrupted or became unstable.

When heartbeat failures persist, IPMC may trigger a controlled shutdown or reboot to protect the system from hardware instability.

This behavior is hardware protection logic and is not typically triggered by traffic load or firewall policy configuration.

Verification steps:

• Verify reboot reason.

Run:

get system status

Expected result:

The output displays the last reboot reason.

Example:

Last reboot reason: power cycle

• Verify system resource utilization.

Run:

get system performance status

Expected result:

CPU and memory utilization remain within normal thresholds.

• Verify memory conserve status.

Run:

diagnose hardware sysinfo conserve

Expected result:

memory conserve mode: off

If conserve mode is enabled, investigate memory pressure conditions.

• Verify hardware memory integrity.

Run:

diagnose hardware sysinfo memory

Expected result:

HardwareCorrupted value remains 0.

Observed result:

HardwareCorrupted: 0 kB

• Verify session table stability.

Run:

diagnose sys session stat

Expected result:

No abnormal session exhaustion or 'memory_tension_drop' values.

Observed result:

memory_tension_drop=0

• Verify hardware interrupt activity.

Run:

diagnose hardware sysinfo interrupt

Expected result:

Interrupt activity remains stable without abnormal hardware interruption patterns.

Special attention should be given to:

• PCI devices.

• NVMe devices.

• HA interfaces.

• Management interfaces.

• Verify system event logs.

Go to Log & Report -> System Events.

Review logs related to:

• IPMC event.

• PCI heartbeat.

• ipmc_sensord.

• Interface status changes.

Expected result:

Identify the sequence of events preceding the reboot.

Observed sequence:

1. IPMC Event generated.

2. Emergency reboot countdown initiated.

3. Interface link-down events generated.

4. ipmc_sensord initiated shutdown.

5. System reboot completed.

Troubleshooting:

If IPMC Event logs continue to appear, perform the following checks:

• Verify hardware health status.

Run:

diagnose hardware deviceinfo nic

Expected result:

All hardware components report normal operational state.

• Verify storage subsystem status.

Review NVMe interrupt activity and disk-related events.

Expected result:

No storage-related hardware errors are present.

• Verify HA heartbeat stability.

If HA is enabled, verify heartbeat interfaces.

Run:

get system ha status

Expected result:

HA heartbeat interfaces remain stable.

• Verify environmental conditions.

Check:

• Power stability.

• Temperature.

• Rack ventilation.

• Power supply status.

• Verify firmware version.

Observed version:

Verify whether known hardware or IPMC-related issues exist for the installed firmware version.

• Collect crash and event logs.

Run:

diagnose debug crashlog read

Expected result:

No recurring hardware panic or kernel crash entries are present.

• Escalate to hardware investigation if symptoms persist.

Repeated PCI heartbeat events may indicate:

1. PCI communication instability.

2. Motherboard issues.

3. Hardware bus instability.

4. Storage subsystem instability.

5. Power-related hardware faults.

Note:

After upgrading to FortiOS v7.4.11 and v7.6.6, FortiGate 70XG units may experience a single unexpected reboot. FortiOS may have detected an event, but the crash logs do not provide sufficient details. System events will show the similar IPMC event as described in this article. 

This is a known bug 1283972. A fix will be available in FortiOS v7.4.13, v7.6.8 and v8.0.2. 

The current workaround is to switch a security level from BIOS. See Technical Tip: Change security level on FortiGate G series models.

A Customer Support Bulletin (CSB) with Bulletin ID CSB-260817-1 has been published regarding this issue BulletinList. For information on how to access the Customer Support Bulletin, see Customer Support Bulletin

Related articles:

• Technical Guide: Identifying the FortiGate reboot reasons and what to do next

• Troubleshooting Tip: How to deal with a Kernel panic

• Technical Tip: Steps to follow when a fan failure appears on FortiGate

• Troubleshooting Tip: How to troubleshoot CLI 'sensor list', 'sensor detail' and 'sensor information' Widget without information

• Troubleshooting Tip: False positive power supply and temperature alerts on FortiGate