Technical Tip: Manual High Availability (HA) restore in a L3 environment via CLI

[FortiKoala]

Description

This article describes how to restore control to the Primary Server(s) using the CLI. For instructions on resuming control using the Administration UI, see the related KB article below.

Scope

FortiNAC.

Solution

Contact the support if assistance is needed.

Establish SSH sessions to all Appliances (2 if NS500/550's). All SSH sessions should be open at once. 

1. Change the Status from 2 to 1 on all appliances.
   
   1. Type: vi /bsc/campusMgr/bin/.networkConfig
   2. Find and change the line from Status=2 to Status=1
   3. Save changes and exit the file (:wq).
   4. Repeat this on all 4 appliances.
2. Shutdown the management processes on all appliances.
   
   1. Type: shutdownNAC
   2. Wait 30 seconds.
   3. Type: shutdownNAC ‐kill
   4. Repeat this on all 4 appliances.
3. (Optional) Restore the DataBase to the Primary from the Secondary Server. If this is not necessary, skip to step 4.
   
   On the Secondary Control Server:
   1. Type: cd /bsc/campusMgr/master_loader/mysql
   2. Type: ydb_dated_backup
   3. Type: cd backup
   4. Locate the latest Database backup file.
   5. Copy database backup to Primary Control Server, Type: 

   scp <filename of backup> cm1/bsc/campusMgr/master_loader/mysql/backup

   On Primary Control Server:
   
   1. Type: cd /bsc/campusMgr/master_loader/mysql
   2. Type: ydb_restore_full_backup <filename of backup>
      
      
4. Start services on the Primary system.
   
   1. On Primary Control Server type: startupNAC
   2. Allow about 10 minutes for the system to start up.

On the Primary Control and Application Server the following commands can be run to monitor logs.

cd /bsc/log

tail -f output.processManager

Look for the following line to appear (may take a minute before it shows up):

<hostname>(Master) Master In Control Idle(false) Total Memory………

5. Start services on the Secondary system (will be in standby).

On the Secondary Control Server, run the following commands:

startupNAC

cd /bsc/logs

tail –f output.processManager

Look for the following line to appear (it may take a few minutes before it shows up):

<hostname>(Slave) Master In Control Idle(false) Total Memory………

6. Log into the UI on the Primary system. 

On the Dashboard screen confirm, the Summary panel shows: Primary Servers as 'Running - In Control'. Secondary Servers as 'Running - Not In Control'.

If the database backup restore, as mentioned in step 3, is not required/necessary, then proceed to step 4 and start the services on the primary appliance.
 

Note:

In later versions of FortiNAC that are running FortiNAC-OS, the editor can be used with the (-N) flag as follows:

vi -N /bsc/campusMgr/bin/.networkConfig

And the database backup file path has changed to: /bsc/backups/database/

Related articles:

Technical Tip: Manually resume control to primary server via CLI

Technical Tip: Restoring Primary Servers after High Availability (HA) failover

[blarochelle]

Think we can break down step 1 to 2 steps.
1. Enter the command: vi -c "%s/Status=2/Status=1/g|wq" /bsc/campusMgr/bin/.networkConfig

2. Repeat this on all 4 appliances