Technical Tip: Moving FortiSOAR HA Node Cluster to New Network

[snikam]

Description	This article describes the process of cloning existing HA nodes to different network segments without breaking clusters.
Scope	FortiSOAR HA Nodes.
Solution	Pre-requisites: All data ingestion schedules and listener-based data ingestions should be stopped. Deactivate Tenant from Master.  For more details check the guide here. Deactivate the FortiSOAR agent if applicable. Note down the existing device UUIDs using the following command and store them: csadm license --get-device-uuid Note down the old IP address for all HA nodes. Suspend cluster. Run the following command on the secondary active or passive node: csadm ha suspend-cluster For more details about suspend-cluster check the guide here. On all cluster nodes, disable PostgreSQL-14, celeryd , celerybeatd to prevent them from starting up post boot:  systemctl disable celeryd celerybeatd postgresql-14 Stop all services on all nodes: csadm services –stop Shutdown all VMs Steps. Steps: Moving HA Nodes to the new network: Move all HA nodes into the new network using the process as per the platform on which FortiSOAR VM(s) are hosted. Set the new static IP address to all nodes in case DHCP is not present. Update /etc/hosts file on all nodes with newly set IPs in case DNS is not present or applicable. This should include self-entry as well as entry for other system(s). Check outbound connectivity with the internet, repo.fortisoar.foritnet.com, and threat intel like VirtusTotal, IP Stack, LDAP / IDP server (if configured), etc. Firewall Rules Reconfigurations: Remove Old Firewall Rules: List down the existing rich rules using the below command:  firewall-cmd --list-rich-rules Copy the rules that contain the old secondary node IPs in Notepad or related app which will be used to remove them one by one. To remove old rules one after the other which contain the old secondary node IPs, use the below command: firewall-cmd --remove-rich-rule='<copy rule here>' --permanent Example: firewall-cmd --remove-rich-rule='rule family="ipv4" source address="10.132.253.161/32" port port="5671" protocol="tcp" accept' --permanent Do this for the other two ports – 5432 and 9200. Reload the firewall using the command firewall-cmd --reload. Add New Firewall Rules: Now Add new rules with new IPs using the below command: firewall-cmd --add-rich-rule='<copy rule here>' --permanent Example: firewall-cmd --add-rich-rule='rulefamily="ipv4" source address="192.168.1.161/32" port port="9200" protocol="tcp" accept' --permanent Do this for the other two ports – 5432 and 9200. Reload the firewall using the command firewall-cmd --reload. Resuming Cluster: First, start all services on the primary node. After services are online on the primary node, start all services on the secondary node. Now enable PostgreSQL-14 on the primary node and start the service: systemctl enable postgresql-14 systemctl start postgresql-14 Refresh device UUID on the primary node, this will update the cluster: csadm license --refresh-device-uuid Now, enable PostgreSQL-14 on the secondary node and start the service: systemctl enable postgresql-14 systemctl start postgresql-14 Refresh device UUID on the secondary node, this will update the cluster: csadm license --refresh-device-uuid On the Secondary node perform the following: Note down new device UUID using the command csadm license --get-device-uuid, Edit file /var/lib/pgsql/fsr-suspend-cluster-backup/.current_node_details, Update the new device UUID in the file by updating the key nodeId. Enable celeryd and celerybeatd services: systemctl enable celeryd celerybeatd systemctl start celeryd celerybeatd Resume Cluster with the following command. The command must be executed from a secondary active or passive node. csadm ha resume-cluster Resume communication with master nodes from tenant node if applicable. Activate tenant/agent if applicable. Resume the data ingestion schedules that were stopped. Resume listener-based ingestions which were stopped. Delete the old replication slot. This step is recommended to free up disk space. On the Primary delete the old replication slot using the command: List slots - csadm ha utils replication-slots list Remove slot - csadm ha utils replication-slots remove --slot-name <slot name with old device UUID>  Delete the old VMs. Troubleshooting: Post start of services, in case the resume command fails to work, then do the following checks  Edit file /var/lib/pgsql/14/data/pg_hba.conf on all nodes and replace the existing IPs with new IPs under FortiSOAR PG HA configuration for the master section as shown in the below screenshot, ignore in case hostnames are used in a given file instead of IPs. Edit file /var/lib/pgsql/14/data/postgresql.conf on all secondary nodes and update the parameter - primary_conninfo by replacing the existing IPs with new IPs, ignore in case hostnames are used in a given file instead of IPs.