|
How to check if the MySQL process is running fine:
Log in to the primary FortiNAC CLI with the username 'admin' and execute the command shown below
FortiNAC # execute enter-shell <----- Execute this command to enter the shell mode.
FortiNAC: $ systemctl status mysqld
mysqld.service - MariaDB database server
Loaded: loaded (/lib/systemd/system/mysqld.service; disabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/mysqld.service.d
Loverride.conf
Active: active (running) since Tue 2024-10-22 14:38:32 GMT: 1 day 19h ago
Process: 4672 ExecStartPost=/usr/bin/mysql-systemd-start post (code=exited, status=0/SUCCESS)
Main PID: 4836 (mariadbd)
Tasks: 36 (limit: 9532)
Memory: 124.5M
CGroup: /system.slice/mysqld.service
The output must say that the MySQL process is active and running.If it says 'failed', check for the error messages using the command shown below.
FortiNAC # diagnose tail mysqld.log
Error Example:
2024-10-23 23:47:34 0 ERROR] Found 1 prepared transactions! It means that server was not shut down properly last time and critical recovery information (
last binlog or tc. log file) was manually deleted after a crash. You have to start server with -- tc-heuristic-recover switch to commit or rollback pending
zransactfons.
2024-10-23 23:47:34 0 [ERROR] Aborting
Procedure:
- Login into the both the primary and the secondary FortiNAC CLI as 'admin' user and check if the Fortinac_Database_backups files are present:
FortiNAC #execute enter-shell
FortiNAC: $ cd /bsc/backups/database
FortiNAC: $ ls -l
- The FortiNAC database file will be saved with the syntax shown here: <database>_<yyyy_mm_dd_hh_mm_ss>.gz
- The FortiNAC database file contains the components and configurations viewed/modified through the Administration UI and the last known state of those components. Everything seen in the Administration UI is kept in the database except for Alarms and Events, Connection Logs and Scan Results. Passwords are encrypted.
- If the error message appeared at 2024-10-23 23:47:34 choose a database file that was generated during the working state for example FortiNAC_DataBase_BackUp_2024_10_20_00_01_43_fnac-ca1.gz which was generated On OCT-20-2024.
- Validate if both the Primary and secondary server contains the same file:
- Validate the files by calculating the hash for the database files as shown below:
FortiNAC: $ sha256sum FortiNAC_DataBase_BackUp_2024_10_24_00_01_01_fnac-ca1.gz
55835bd6c1495a4021e5c8a8813b58223ef64e483059ee16481cb351fd524d0e FortiNAC_DataBase_BackUp_2024_10_24_00_01_01_fnac-ca1.gz
- If both the hash values are the same, it represents that both the files are the same.
- Failover of the control from primary to secondary FortiNAC instance:
- If the primary server is in control, perform the failover to the secondary by manually shutting down the processes on Primary FortiNAC.
- To identify which server is in control use the command shown below on both the instances.
FortiNAC: $tail –F /bsc/logs/output.processManager | grep "In Control Idle"
- The following message shown indicate that the primary is in control:
Primary Server: (Primary) Primary In Control Idle(false)
Secondary Server: (Secondary) Primary In Control Idle(false)
- If the primary is in control initiate the failover to the secondary as shown below
FortiNAC: $ shutdownCampusManager
FortiNAC: $ logs
FortiNAC: $ tail -f output.processmanager | grep "In Control Idle"
- If the failover is successful the following message can be seen while tailing the 'output.processmanager' log on both the FortiNAC instances
Primary Server: (Primary) secondary In Control Idle(false)
Secondary Server: (Secondary) secondary In Control Idle(false)
- Post failover restore the database file on secondary server:
- Restore the database using the command shown below:
FortiNac: $ydb_restore_full_backup FortiNAC_DataBase_BackUp_2024_10_20_00_01_43_fnac-ca1.gz
- Wait for 10-20 minutes for the GUI to be accessible again and then it will be possible to log in with an admin username/password.
- Rebuild MySQL database on the primary FortiNAC after failing over the control to the secondary FortiNAC server:
- Rebuilt the MySQL database using the command shown below
FortiNAC: $ exit
FortiNAC # execute sql initialize
- Verify the SQL status using the command 'systemctl status mysqld' as shown before from the shell mode.
If errors persist, record the timestamp during the SQL re-initialization, gather the relevant log files, and open a TAC case.
For Gathering the log files refer to this Community Article:
Technical Tip: How to get a debug log report from FortiNAC-CA or FortiNAC-Manager
- Restore the database on the primary server and resume control on primary server:
- Restore the database using the command shown below
FortiNAC: $ydb_restore_full_backup FortiNAC_DataBase_BackUp_2024_10_20_00_01_43_fnac-ca1.gz
- After restoring the database, the control can be resumed back to the primary FortiNAC server via GUI or through CLI.
- GUI Method: On GUI browse to Dashboard -> Summary, and select 'resume control' under the primary server.
- CLI Method: On the CLI of the secondary FortiNAC server shut down the processes using the 'shutdownCampusManager' shell command as shown before in step 3.
Wait for a few minutes for the GUI to be accessible again.
|
