Technical Tip: Manual High Availability (HA) restore in a L3 environment via CLI

[FortiKoala]

Description

This article describes how to restore control to the Primary Server(s) using the CLI. For instructions on resuming control using the Administration UI, see the related KB article below.

Scope

FortiNAC v6 and above.

Solution

Contact the support if assistance is needed.

Establish SSH sessions to all 4 Appliances (2 if NS500/550's). All SSH sessions should be open at once. 

1. Change the Status from 2 to 1 on all 4 appliances.
   
   1. Type cd /bsc/campusMgr/bin.
   2. Type vi .networkConfig.
   3. Change Status=2 to Status=1.
   4. Save changes and exit the file.
   5. Repeat this on all 4 appliances.
2. Shutdown the management processes on all appliances.
   
   1. Type shutdownNAC.
   2. Wait 30 seconds.
   3. Type shutdownNAC ‐kill.
   4. Repeat this on all 4 appliances.
3. (Optional) Restore the DataBase to the Primary from the Secondary Server. If this is not necessary, skip to step 4.
   
   On the Secondary Control Server:
   1. Type cd /bsc/campusMgr/master_loader/mysql.
   2. Type ydb_dated_backup.
   3. Type cd backup.
   4. Locate the latest Database backup file.
   5. Copy database backup to Primary Control Server (this command should work): Type: 

   scp <filename of backup> cm1/bsc/campusMgr/master_loader/mysql/backup

   On Primary Control Server:
   
   1. Type cd /bsc/campusMgr/master_loader/mysql.
   2. Type ydb_restore_full_backup <filename of backup>.
      
      
4. Start services on the Primary system.
   
   1. On Primary Control Server type startupNAC.
   2. Allow about 10 minutes for the system to start up.

On the Primary Control and Application Server the following commands can be run to monitor logs.

cd /bsc/log

tail -f output.processManager

Look for the following line to appear (may take a minute before it shows up):

<hostname>(Master) Master In Control Idle(false) Total Memory……….

5. Start services on the Secondary system (will be in standby).

On the Secondary Control Server, run the following commands:

startupNAC

cd /bsc/logs

tail –f output.processManager.

Look for the following line to appear (it may take a few minutes before it shows up):

<hostname>(Slave) Master In Control Idle(false) Total Memory……….

6. Log into the UI on the Primary system. 

On the Dashboard screen confirm, the Summary panel shows: Primary Servers as 'Running - In Control'. Secondary Servers as 'Running - Not In Control'.

If the database backup restore, as mentioned in step 3, is not required/necessary, then proceed to step 4 and start the services on the primary appliance.
 

Related article:

Technical Tip: Restoring Primary Servers after High Availability (HA) failover

[blarochelle]

Think we can break down step 1 to 2 steps.
1. Enter the command: vi -c "%s/Status=2/Status=1/g|wq" /bsc/campusMgr/bin/.networkConfig

2. Repeat this on all 4 appliances