Troubleshooting Tip: Potential Reason for FortiGate intermittently rebooting without major hardware or software issue
| Description | This article describes the chronological steps to investigate an issue where the FortiGate was observed being rebooted every 2 hours, although the Hardware Quick Inspection Package (HQIP) test on the FortiGate passes all tests, and there are no software bugs observed. |
| Scope | FortiOS. |
| Solution | In a scenario where the FortiGate randomly went unresponsive and recovered without any performed actions, it is always recommended to check the following information:
While there could be many reasons that could cause a reboot on the FortiGate, this article focuses on the behavior where FortiGate was being rebooted every 2 hours due to misconfiguration on Automation Stitch. In the FortiGate command line, the information would be illustrated as follows for symptoms that it is being rebooted by an administrator or triggered under certain conditions:
Refer to the following article for different types of reboots on FortiGate: Technical Guide: Identifying the FortiGate reboot reasons and what to do next.
Besides the reboot reason, the crashlog indicates that the FortiGate interfaces were brought down by the process_name 'autod':
From the information above, it is possible to narrow down to the abnormality caused by the autod daemon. The autod daemon is the daemon that handles Automation Stitches functionality.
Connecting a laptop or host directly to the FortiGate console port is another way to identify what was happening during the time of the incident. In this incident, it has been observed that there were no kernel crashes that occurred before the system was instructed to reboot. This scenario usually indicates a warm reboot:
In the System event logs, it is also possible to identify the root cause of the reboot if it's triggered under the same condition explained in this article. The following example illustrates the system event where automation stitch 'IPS_AV_DB' is being triggered, and it follows with the system event that states 'User rebooted the device from autod':
Navigate to 'Security Fabric -> Automation' to check on the configuration of the automation stitch named 'IPS_AV_DB'. It has been confirmed that an automation stitch was configured to reboot the FortiGate when the antivirus and IPS DB gets updated:
Deleting or modifying the action on this automation stitch resolves the issue of the FortiGate being rebooted every 2 hours. The reason for the duration is that the default scheduled update settings on the FortiGate are being set to every 2 hours. Technical Tip: How to check most updated security version database on FortiGate
The default auto-update schedule has been changed to daily from FortiOS v7.6.5 and above under bug ID 1204277: Changes in default behavior. If this misconfiguration is done on FortiOS v7.6.5 and above, the FortiGate will be observed being rebooted on a daily basis when FortiGate performs IPS and antivirus database updates.
Automation stitches is a powerful tool to automate certain information collection, perform certain actions, and/or trigger an alert to the administrator when a certain scenario is being detected. Hence, it is important to understand the impact of an automation stitch before configuring the relevant settings.
Related documents: Technical Tip: Short list of processes on the FortiGate Technical Tip: Use FortiGate automation stitches for alert emails Technical Tip: Creating automation stitches Technical Tip: How to configure an Automation Stitch to execute a packet capture in a desired time Technical Tip: How to configure Automation stitch for Downstream FortiGate and confirm it is working Technical Tip: Using automation stitches to Enable/Disable IPV4 Firewall Policy based on SD-WAN logs Technical Tip: How FortiGate Trigger Automation-stitch with Multiple Events |
