Troubleshooting Tip: How to automate useful command to troubleshoot intermittent HA synchronization issue using FortiLightHouse
Description
This article describes methods to troubleshoot intermittent HA (High Availability) synchronization issues in FortiGate devices using FortiLightHouse.
Scope
FortiGate.
Solution
FortiLightHouse CLI (FlhCli) is a monitoring utility designed to collect critical outputs from FortiGate firewalls. It gathers the necessary debug information to assist Fortinet TAC and Engineering teams in performing in-depth analysis.
Download the FlhCli tool on the system from the link below:
Once the binary file has been downloaded, open the Command Prompt by typing the commands in the search bar. Navigate to the download directory from the Command Prompt & execute flhcli.exe to start the application.
Run the command flhcli.exe setup to create the required XML configuration file.
A monitoring script is required to run against the predefined device.
A sample HA script. It can be adjusted as per the requirement.
<flhscript version="1">
<parameters name="default">
<list name="status">
<query type="pids">
<option name="process">cpu</option>
</query>
</list>
</parameters>
<settings name="default">
<ics>100ms</ics>
<cycleDuration>60s</cycleDuration>
</settings>
<cycle name="performance" parameters="default" settings="default">
<command context="global">get system ha status</command>
<command context="global">print global</command>
<command context="global">diagnose sys ha checksum cluster</command>
<command context="global">diagnose sys ha checksum cached global</command>
<command context="global">diag sys ha checksum cached root</command>
<command context="global">diagnose sys ha hasync-stats all</command>
<command context="global">diagnose sys ha sync-stats</command>
<command context="global">diagnose sys ha hasync-stats most-recent</command>
<command context="global">diagnose sys ha hasync-stats most-time</command>
<command context="global">diagnose sys ha hasync-stats most-time</command>
<command context="global">diagnose debug console timestamp enable</command>
<command context="global">diagnose debug application hasync -1</command>
<command context="global">diagnose debug application hatalk -1</command>
<command context="global">diagnose debug enable</command>
</cycle>
</flhscript>
Ensure that the monitoring script and the flhcli XML files are stored in the same directory.
The script output is saved in a file named HUBFGT_monitor.txt. The script is configured with a 30-second cycle interval, which can be adjusted as required.
Execute the command below to run the script.
flhcli -force-pty -redirect <<<device_name>>>_monitor --rotate-age 4h --name <<<device_name>>> cmd xml --xml <<script name>> --cycle performance
For example:
flhcli -force-pty -redirect HUBFGT_monitor --rotate-age 4h --name HUBFGT cmd xml --xml IPSEC_VPN.xml --cycle performance
Output can be viewed from the file HUBFGT_monitor:
