Technical Note: CSTN: 00047 - Adding multiple disks and partitioning disks to your FortiSOAR VM, and and recovering FortiSOAR data from a VM

[Andy_G]

Description

FortiSOAR™ 6.0.0 onwards multiple disks are supported for the FortiSOAR™ installation. Having multiple disks for the FortiSOAR™ installation has an advantage that if your FortiSOAR™ system crashes, you can detach the disks that contain data and recover data. The procedure for recovering data is present in the Recovering Data section later in the article. The procedure for extending existing disks is present in the FortiSOAR™ documentation in the Deployment Troubleshooting chapter of the "Deployment Guide."

You can add three more disks to your Virtual Machine (VM) and create separate Logical Volume Management (LVM) partitioning for PostgreSQL and Elasticsearch data.

For example, you have added the following new disks:

• /dev/sdb: Recommended to have a thin provisioned disk for PostgreSQL data whose disk size is 500GB. 
• /dev/sdc: Recommended to have a thin provisioned disk for Elasticsearch data whose disk size is 150GB.
• /dev/sdd: Recommended to have a thin provisioned disk for FortiSOAR™ RPM data whose disk size is 20GB.

Solution

Partitioning the disks

To partition the /dev/sdb, which is the disk for PostgreSQL data, run the following commands as a root user:

1. pvcreate /dev/sdb
2. vgcreate vgdata /dev/sdb
3. mkdir -p /var/lib/pgsql
4. lvcreate -l 100%VG -n relations vgdata
5. mkfs.xfs /dev/mapper/vgdata-relations
6. mount /dev/mapper/vgdata-relations /var/lib/pgsql
7. echo "/dev/mapper/vgdata-relations /var/lib/pgsql xfs defaults 0 0" >> /etc/fstab

To partition the /dev/sdc, which is the disk for Elasticsearch data, run the following commands as a root user:

1. pvcreate /dev/sdc
2. vgcreate vgsearch /dev/sdc
3. mkdir -p /var/lib/elasticsearch
4. lvcreate -l 100%VG -n search vgsearch
5. mkfs.xfs /dev/mapper/vgsearch-search
6. mount /dev/mapper/vgsearch-search /var/lib/elasticsearch
7. echo "/dev/mapper/vgsearch-search /var/lib/elasticsearch xfs defaults 0 0" >> /etc/fstab

To partition the /dev/sdd, which is the disk for FortiSOAR™ RPM data, run the following commands as a root user:

1. pvcreate /dev/sdd
2. vgcreate vgapp /dev/sdd
3. mkdir -p /opt
4. lvcreate -l 100%VG -n csapps vgapp
5. mkfs.xfs /dev/mapper/vgapp-csapps
6. mount /dev/mapper/vgapp-csapps /opt
7. echo "/dev/mapper//vgapp-csapps  /opt    xfs    defaults    0 0" >> /etc/fstab

Recovering data

Note: Commands for recovery of data must be run as a root user.

Following is the procedure for recovering data from the disks:

1. Deploy the recovery VM that has the same version of FortiSOAR™ installed (OVA or AMI) and power it ON.
2. In the /etc/fstab file, comment out the lines that contain the word vgdata or vgapp.
3. Rename the vgdata and vgapp volume groups using the following command:
   vgrename vgdata old_vgdata
vgrename vgapp old_vgapp
4. Stop all FortiSOAR™ services using the following command:
   csadm services --stop
5. Run the umount /var/lib/pgsql/ && umount /opt command.
6. Deactivate the volume group using the following command:
   vgchange -a n old_vgdata old_vgapp
7. Find out which disks contain the vgdata and vgapp volume groups using 'pvs' command.
   For example, if vgdata is on /dev/sdb and vgapp is on /dev/sdd, you require to skip these disks from lvm scanning. You can skip the disks from lvm scanning by adding the skip filter in /etc/lvm/lvm.conf file, by performing the following steps:
   1. Open the etc/lvm/lvm.conf file using the vi /etc/lvm/lvm.conf command.
   2. In the "devices {" section in the lvm.conf file, add the following line:
      filter = ["r|/dev/sdb|", "r|/dev/sdd|"]
8. Stop the old VM and attach the existing PostgreSQL and RPM disks from the old VM to the recovery VM.
   The PostgreSQL disk will have the size of 150GB and the RPM disk will have the size of 10GB.
9. Run the vgs command, which should display the vgdata and vgapp volume groups.
10. In the /etc/fstab file, uncomment the lines that contain the word vgdata or vgapp that we had commented out in step 2.
11. Reboot your recovery VM.
12. Truncate the envc and cascade tables using the following command:
    psql -U cyberpgsql -d das -c "truncate envc cascade;"
13. Update the cluster table using the following shell script. You can also create a temporary shell script using the following contents and run the same. For example, sh temp_script_for_cluster_table_updation.sh:
    hardware_key=`csadm license --get-hkey` 

    current_hostname=`hostname`

    #First findout the number of nodes available in cluster table

    number_of_nodes_in_cluster_table=`psql -U cyberpgsql -d das -tAc "select COUNT(*) from cluster;"`

    if [ $number_of_nodes_in_cluster_table -eq 1 ]; then

        # Only single node is available in cluster, hence directly update the nodeid.

        psql -U cyberpgsql -d das -c "UPDATE cluster SET nodeid='${hardware_key}';"

        csadm ha set-node-name $current_hostname

    elif [ $number_of_nodes_in_cluster_table -gt 1 ]; then

        # More than one node is available. Now update the nodeid where nodename in cluster table matches with current hostname

        psql -U cyberpgsql -d das -c "UPDATE cluster SET nodeid='${hardware_key}' where nodename='${current_hostname}';"

    else

        echo "Not able to update the cluster table"

    fi

14. Important: This step is applicable only to the 6.x series:
    Change the redis password using the following commands:
    #findout current redis password
redis_password=`grep -e "^\s*requirepass\s\+" /etc/redis.conf | cut -d' ' -f2`

#enycrypt the redis password
redis_password=`/opt/cyops-auth/.env/bin/python /opt/cyops/configs/scripts/manage_passwords.py --encrypt $redis_password`

/opt/cyops-auth/.env/bin/python /opt/cyops/configs/scripts/confUtil.py -f /opt/cyops/configs/database/db_config.yml -k "redis_password" -v $redis_password

/opt/cyops-auth/.env/bin/python /opt/cyops/configs/scripts/confUtil.py -f /opt/cyops/configs/database/db_config.yml -k "redis_password_primary" -v $redis_password
15. Change the rabbitmq password using the following commands:
    systemctl start rabbitmq-server
rabbitmq_password=`grep "cyops.rabbitmq.password" /opt/cyops/configs/rabbitmq/rabbitmq_users.conf | cut -d"=" -f2`
rabbitmqctl change_password cyops $rabbitmq_password
16. Change the elasticsearch password using the following commands:
    elasticsearch_password=`csadm license --get-hkey` 

printf $elasticsearch_password | /usr/share/elasticsearch/bin/elasticsearch-keystore add "bootstrap.password" -f
elasticsearch_password=`/opt/cyops-auth/.env/bin/python /opt/cyops/configs/scripts/manage_passwords.py --encrypt $elasticsearch_password`

/opt/cyops-auth/.env/bin/python /opt/cyops/configs/scripts/confUtil.py -f /opt/cyops/configs/database/db_config.yml -k "secret" -v $elasticsearch_password
17. Clear the API cache using the following commands. If any command fails, rerun that command:
    systemctl start redis
rm -rf /opt/cyops-api/app/cache/prod/
cd /opt/cyops-api
echo "Clear API cache"
sudo -u nginx php app/console cache:clear --env=prod --no-interaction
18. Refresh the Keys using the following command:
    sudo csadm certs -l
19. Update the system using the following command:
    cd /opt/cyops-api/
echo "System Update"
sudo -u nginx php app/console cybersponse:system:update -la --env=prod --force
20. Restart the services using the following command:
    echo "Restarting Services dependent on keys..."
sudo csadm services --restart
21.  Reindex elasticsearch using the following command:
    sudo -u nginx php /opt/cyops-api/app/console cybersponse:elastic:create --env=prod
22. (Optional) If you do not remember the FortiSOAR™ UI password of your old instance and want to reset it to the default, which is 'changeme' for non-AWS instances and the 'instance_id' for AWS instances, run the following command:
    /opt/cyops-auth/.env/bin/python -c "import sys; sys.path.append(\"/opt/cyops-auth\"); import utilities.reset_user as reset_user; reset_user.start()"
23. Redeploy your FortiSOAR™ license.