Technical Tip: FortiGate-6000/7000 Chassis health check commands

[ssudhakar]

Description
This article describes basic commands that is recommended to run in order to check the health of the system.

Scope
For FortiGate-6k-7k .

Solution

# get system status

- These commands gives the information about firmware, build, HA mode ,config-sync and FPC master.

# Diagnose load-balance status

- This command gives the information on which FPC/FPM is the master.
- The status message shows 'Running'.
- In case of any error, the status message shows 'waiting for data heartbeat', 'waiting for configuration sync' etc.

# get system performance status

- A FortiGate that is idle will look like: CPU states: 0% user 0% system 0% nice 100% idle.
- However, if the network is running slow it will be something like:

• CPU states: 1% user 98% system 0% nice 1% idle.

- Memory: gives the info about total/used/free space.The memory usage should not be too high. If >70%, contact support.

# get sys ha status

- This command gives information about Master and slave status like whether the cluster is In-sync, same vlan id, cluster uptime, and ha history. 

- Check - HA Health Status: OK

- Check - MONDEV stats: the 'up/down'. This state should be same on both chassis.

- In case the cluster is of out-of-sync or any of the check fails, contact support.

# diagnose sys confsync status

- Check to see if all blades are in sync (in_sync=1).
- In case of any conf sync issue, the status will show in_sync=0.

# diagnose sys ha dump-by group

- best_hbdev=ha2 gives you information that the communication is over ha2 interface. best_hbdev, dp_rsync_hbdev, slave_fim_dp_rsync_hbdev should select real ports, none of them should be 'NA'.

- It gives information about the number of Active worker blades in a cluster, uptime and reset_cn.

- The active workers should be same on both the chassis.

- link_failure, pingsvr_failure, active_worker should be same.

- Master’s chassis flag should be “1”, Slave chassis flag should be '0'. During upgrade process, the forced master will be set to “3”. Once the upgrade is finished, it will be set to 1. 

# diagnose sys confsync showcsum | grep "SN\|^all"

- Each blade will print out 4 lines of checksum.

- 1st-2nd lines of checksum should be same on all blades on both Master and Slave chassis.

- 3rd-4th lines of checksum should be same within the chassis.

Best Practices during Chassis-upgrade.

1) Perform these health-check commands before and after upgrade to make sure all blades are in running and sync status.

2) When performing step-by-step upgrade, always make sure all blades are up and in sync after each FOS upgrade step before proceeding to next FOS upgrade.

3) Take backup of the config file and it is always preferred to  have console connection and physical access to the device during upgrade window.

Related Articles

Technical Tip: How to find the config difference between blades in 6K/7K Chassis using 'diagnose sys...

Troubleshooting Tip: FortiGate 7000 Series blade config synchronization issues (confsync)