Technical Note: FortiSOARâ„¢ Performance Benchmarking for v6.4.4
Description
This document details the performance benchmark tests conducted in Fortinet labs. The performance benchmarking tests were performed on FortiSOARâ„¢ version 6.4.4 Build 3164.
Solution
Note: Once this test is completed, refer to the pre-test conditions before starting a new test.
This document details the performance benchmark tests conducted in Fortinet labs. The performance benchmarking tests were performed on FortiSOARâ„¢ version 6.4.4 Build 3164.
Solution
The objective of this performance test is to measure the time taken to create alerts in FortiSOARâ„¢, and complete the execution of corresponding playbooks on the created alerts in the following cases:
- Single-node FortiSOARâ„¢ appliance
- Cluster setup of FortiSOARâ„¢
The data from this benchmark test can help you in determining your scaling requirements for a FortiSOARâ„¢ instance to handle the expected workload in your environment.
Single Invocation Test for the single-node FortiSOARâ„¢ appliance
Environment
FortiSOARâ„¢ Virtual Appliance Specifications
| Component | Specifications |
| CPU | 8 CPUs |
| Memory | 32 GB |
| Storage | 250 GB virtual disk, with IOPS 2400, attached to an AWS Instance. |
Operating System Specifications
| Operating System | Kernel Version |
| CentOS 7 | 3.10.0-1160.6.1.el7.x86_64 |
External Tools Used
| Tool Name | Version |
| Zabbix | 4.2.1 |
| Internal Script to gather data |
Pre-test conditions on both the standalone FortiSOARâ„¢ machine and the FortiSOARâ„¢ High Availability (HA) cluster
At the start of each test run -
- The test environment contained zero alerts.
- The test environment contained only the FortiSOARâ„¢ built-in connectors such as IMAP, Utilities, etc.
- The system playbooks were deactivated such as, alert assignment notification, sla calculation,etc, and there were no running playbooks.
- The playbook execution logs were purged.
- Configured tunables as follows:
- Changed celery workers to 16
- Elastic heaps size to 8GB
- Increased PostgresSQL Shared memory to 2048MB and worker_mem to 64MB
- Disable playbook priority. To disable playbook priority, edit the /opt/cyops-workflow/sealab/sealab/config.ini file and set the ENABLE PRIORITY parameter to false. Then restart the
uwsgi and celeryd services, using the following command:
# systemctl restart celeryduwsgi
Notes on the tests performed
In a production environment the following factors might vary, which could affect the observations:
- The size of the alert data.
Note: For all the above tests, the average size of an alert created in FortiSOARâ„¢ is 5KB. - The number of playbooks that are being executed in parallel for each alert. For example, system playbooks for notification or triage/investigate playbooks.
- The number of steps in each playbook.
- The network bandwidth, especially for outbound connections, to applications such as VirusTotal.
Test setup for the single-node FortiSOARâ„¢ appliance
This test has been invoked on a standalone FortiSOARâ„¢ machine.
Tests performed
Test 1: Perform Ingestion in FortiSOARâ„¢ using the FortiSIEM Ingestion Playbook
Description of the Test
This test is executed by manually triggering FortiSIEM Ingestion playbook that creates alerts in FortiSOARâ„¢.
Steps followed
- Created the alerts using the FortiSIEM Ingestion playbook. The JSON for the sample playbooks that have been used are attached with this article (Test_1_Info_Json_Files.zip) so that you can run the same tests in your environment to see the performance in your version/hardware platforms. Or, if you want to do some additions that are specific to your environment, you can also tweak the existing playbooks.
- Once the alerts were created, measured the total time taken to create all the alerts in FortiSOARâ„¢.
Observations
The data in the following table outlines the number of alerts ingested and the total time taken to ingest those alerts.
Single Invocation Test run on a single-node FortiSOARâ„¢ appliance
| Number of alerts created in FortiSOARâ„¢ | Total time (in seconds) taken to create all alerts in FortiSOARâ„¢ | Total number of playbooks executed in FortiSOARâ„¢ |
| 1 | 0.32 | 1 |
| 5 | 0.47 | 1 |
| 10 | 0.67 | 1 |
| 25 | 1.69 | 1 |
| 50 | 2.66 | 1 |
| 100 | 6.13 | 1 |
Test 2: Perform Ingestion in FortiSOARâ„¢ using the FortiSIEM Ingestion Playbook and after the alerts are created execute an "Extraction" playbook
Description of the Test
This test is executed by manually triggering FortiSIEM Ingestion playbook that creates alerts in FortiSOARâ„¢. Once the alerts are created in FortiSOARâ„¢, an "Extraction" playbook is triggered and the total time taken for all the extraction playbooks to complete their execution is calculated.
Steps followed
- Created the alerts using the FortiSIEM Ingestion playbook.
- Once the alerts are created, the "Extraction" playbooks are triggered. The JSON for the sample playbooks that have been used are attached with this article (Test_2_Info_Json_Files.zip) so that you can run the same tests in your environment to see the performance in your version/hardware platforms. Or, if you want to do some additions that are specific to your environment, you can also tweak the existing playbooks.
The playbooks perform the following steps: - Declares variables using the "Set Variable Step".
- Updates the existing indicator list using mapping.
- Retrieves indicators from the source data of the alert.
- Creates indicators in the "Indicators" Module.
- Link alerts to the indicators.
- Update Alert State.
Observations
The data in the following table outlines the number of alerts ingested, the total time taken to ingest those alerts, and the total time taken for all the triggered playbooks to complete their execution.
Single Invocation Test run on a single-node FortiSOARâ„¢ appliance
| Number of alerts created in FortiSOARâ„¢ | Total time (in seconds) taken to execute all the playbooks | Total number of playbooks executed in FortiSOARâ„¢ |
| 1 | 1.62 | 2 |
| 5 | 2.71 | 6 |
| 10 | 3.57 | 16 |
| 25 | 6.35 | 37 |
| 50 | 11.40 | 73 |
| 100 | 23.11 | 144 |
Note: Once this test is completed, refer to the pre-test conditions before starting a new test.
Test 3: Perform Ingestion in FortiSOARâ„¢ using the FortiSIEM Ingestion Playbook and after the alerts are created execute "Extraction" and "Enrichment" playbooks
Description of the Test
The test was executed using an automated testbed that starts FortiSIEM ingestion which in turn created alerts in FortiSOARâ„¢. Once the alerts are created in FortiSOARâ„¢, "Extraction" and "Enrichment" playbooks are triggered and the total time taken for all the extraction and enrichment playbooks to complete their execution is calculated.
Important: The setup for this test is exactly the same, however this test additionally requires the "VirusTotal" connector to be configured.
Steps followed
- Created the alerts using the FortiSIEM Ingestion playbook.
- Once the alerts are created, the "Extraction" playbooks are triggered. The JSON for the sample playbooks that have been used are attached with this article (Test_3_Info_Json_Files.zip) so that you can run the same tests in your environment to see the performance in your version/hardware platforms. Or, if you want to do some additions that are specific to your environment, you can also tweak the existing playbooks.
The playbooks perform the following steps:- Declares variables using the "Set Variable Step".
- Updates the existing indicator list using mapping.
- Retrieves indicators from the source data of the alert.
- Creates indicators in the "Indicators" Module.
- Link alerts to the indicators.
- Update Alert State.
- Once the indicators were extracted, the "Enrichment" playbooks are triggered and they perform the following steps: Once the indicators were extracted, the "Enrichment" playbooks are triggered and they perform the following steps:
- Matches the IP in an internal subnet through the "Utilities" subnet.
- Validates whether the IP is Private or Public.
- Performs enrichment using the "Utilities" connector, if the IP is "Private".
Performs enrichment using the "VirusTotal" connector, if the IP is "Public". - Updates the indicator status based on the IP’s vulnerability.
- Updates the state of the indicator State.
Observations
The data in the following table outlines the number of alerts ingested, the total time taken to ingest those alerts, and the total time taken for all the triggered playbooks to complete their execution.
Single Invocation Test run on a single-node FortiSOARâ„¢ appliance
| Number of alerts created in FortiSOARâ„¢ | Total time (in seconds) taken to execute all the playbooks | Total number of playbooks executed in FortiSOARâ„¢ * |
| 1 | 4.93 | 4 |
| 5 | 6.28 | 16 |
| 10 | 11.53 | 31 |
| 25 | 25.87 | 76 |
| 50 | 47.52 | 151 |
| 100 | 1 minute 34 seconds | 301 |
Note: Once this test is completed, refer to the pre-test conditions before starting a new test. Also, note that enrichment of playbooks makes API calls over the Internet, and the times mentioned in this table to execute playbooks is inclusive of this time.
Analysis of CPU load average utilization when the test run was in progress on the appliance:
Using the system resources specified in the "Environment" and "Pre-Test Conditions" sections, it was observed that while the "Sustenance Test" was running, the "Read" Wait for the PostgreSQL disk averaged around 1 millisecond, with the maximum wait of 3 milliseconds and the minimum wait of 0 milliseconds. The "Write" Wait for the PostgreSQL disk averaged around 1 millisecond, with the maximum wait of 5.5 milliseconds and the minimum wait of 1 millisecond.
Using the system resources specified in the "Environment" and "Pre-Test Conditions" sections, it was observed that while the "Sustenance Test" was running, the "Read" Wait for the PostgreSQL disk averaged around 0 millisecond. The "Write" Wait for the PostgreSQL disk averaged around 8 milliseconds, with the maximum wait of 25 milliseconds and the minimum wait of 1 millisecond.
Sustained Invocation Test for the single-node FortiSOARâ„¢ appliance
Description
A sustenance test was also conducted with the configuration as defined in "Test 2", i.e., the test is executed by manually triggering FortiSIEM Ingestion playbook that creates alerts in FortiSOARâ„¢. Once the alerts are created in FortiSOARâ„¢, an "Extraction" playbook is triggered and the total time taken for all the extraction playbooks to complete their execution is calculated.
Number of alerts: 100/min
Duration: 12 hours
Playbooks configured: As defined in Test 2 comprising of "Ingestion" and "Indicator Extraction" playbooks.
Total number of playbooks executed: 72720
Results
The system performed well under the sustained load. All 72000 alerts were successfully ingested and all the extraction playbooks were successfully completed without any queuing.
Graphs
The following graphs are plotted for the vital statistics for the system that was under test during the period of the test run.
CPU Load Average Utilization Graph
Using the system resources specified in the "Environment" and "Pre-Test Conditions" sections, it was observed that while the "Sustenance Test" was running the CPU utilization was normal and the performance of the system did not get impacted.
Memory Utilization Graph
Analysis of memory utilization when the test run was in progress on the appliance:Using the system resources specified in the "Environment" and "Pre-Test Conditions" sections, it was observed that while the "Sustenance Test" was running the Memory utilization was around 10GB.
Redis PB Queue Count Graph
Analysis of Redis PB Queue Count when the test run was in progress on the appliance:
Using the system resources specified in the "Environment" section and tunables configured as mentioned in the "Pre-Test Conditions" section, it was observed that while the "Sustenance Test" the redis_pb_queue was at 0. This means that no playbooks were getting queued and all the required playbooks associated with the alerts were getting completed, i.e., alerts were created and its indicators were extracted and enriched in a minute.
IO Wait Graph
Analysis of IO Wait when the test run was in progress on the appliance:
Using the system resources specified in the "Environment" and "Pre-Test Conditions" sections, it was observed that while the "Sustenance Test" was running the IO Wait time was normal, with the average IO Wait being around 1% of the CPU idle time.
Read/Write IO Wait Graph for ElasticSearch
Analysis of Read/Write IO Wait for ElasticSearch when the test run was in progress on the appliance:
Using the system resources specified in the "Environment" and "Pre-Test Conditions" sections, it was observed that while the "Sustenance Test" was running the "Read" Wait for the ElasticSearch disk was almost 0 milliseconds. The "Write" Wait for the ElasticSearch disk averaged around 30 milliseconds, with the maximum wait of 40 milliseconds and the minimum wait of 1 millisecond.
Read/Write IO Wait Graph for PostgreSQL
Analysis of Read/Write IO Wait for PostgreSQL when the test run was in progress on the appliance:
Using the system resources specified in the "Environment" and "Pre-Test Conditions" sections, it was observed that while the "Sustenance Test" was running, the "Read" Wait for the PostgreSQL disk averaged around 1 millisecond, with the maximum wait of 3 milliseconds and the minimum wait of 0 milliseconds. The "Write" Wait for the PostgreSQL disk averaged around 1 millisecond, with the maximum wait of 5.5 milliseconds and the minimum wait of 1 millisecond.
Single Invocation Test for the High Availability (HA) active-active cluster of two FortiSOARâ„¢ nodes
Test setup for the HA active-active cluster of two FortiSOARâ„¢ nodes
This test has been invoked the following setup:
- Cluster of two FortiSOARâ„¢ machines that are joined in the Active-Active state using the FortiSOARâ„¢ HA feature
- The machines that form the HA cluster must be in the same network subnet.
Tests performed
Test 1: Perform Ingestion in FortiSOARâ„¢ using the FortiSIEM Ingestion Playbook
Description of the Test
This test is executed by manually triggering FortiSIEM Ingestion playbook that creates alerts in FortiSOARâ„¢.
Steps followed
- Created the alerts using the FortiSIEM Ingestion playbook. The JSON for the sample playbooks that have been used are attached with this article (Test_1_Info_Json_Files.zip) so that you can run the same tests in your environment to see the performance in your version/hardware platforms. Or, if you want to do some additions that are specific to your environment, you can also tweak the existing playbooks.
- Once the alerts were created, measured the total time taken to create all the alerts in FortiSOARâ„¢.
Observations
The data in the following table outlines the number of alerts ingested and the total time taken to ingest those alerts.
Single Invocation Test run on a two-node active-active FortiSOARâ„¢ cluster
| Number of alerts created in FortiSOARâ„¢ | Total time (in seconds) taken to create all alerts in FortiSOARâ„¢ | Total number of playbooks executed in FortiSOARâ„¢ |
| 1 | 0.80 | 1 |
| 5 | 0.48 | 1 |
| 10 | 0.70 | 1 |
| 25 | 1.45 | 1 |
| 50 | 2.69 | 1 |
| 100 | 6.36 | 1 |
Note: Once this test is completed, refer to the pre-test conditions before starting a new test.
Test 2: Perform Ingestion in FortiSOARâ„¢ using the FortiSIEM Ingestion Playbook and after the alerts are created execute an "Extraction" playbook
Description of the Test
This test is executed by manually triggering FortiSIEM Ingestion playbook that creates alerts in FortiSOARâ„¢. Once the alerts are created in FortiSOARâ„¢, an "Extraction" playbook is triggered and the total time taken for all the extraction playbooks to complete their execution is calculated.
Steps followed
- Created the alerts using the FortiSIEM Ingestion playbook.
- Once the alerts are created, the "Extraction" playbooks are triggered. The JSON for the sample playbooks that have been used are attached with this article (Test_2_Info_Json_Files.zip) so that you can run the same tests in your environment to see the performance in your version/hardware platforms. Or, if you want to do some additions that are specific to your environment, you can also tweak the existing playbooks.
The playbooks perform the following steps: - Declares variables using the "Set Variable Step".
- Updates the existing indicator list using mapping.
- Retrieves indicators from the source data of the alert.
- Creates indicators in the "Indicators" Module.
- Link alerts to the indicators.
- Update Alert State.
Observations
The data in the following table outlines the number of alerts ingested, the total time taken to ingest those alerts, and the total time taken for all the triggered playbooks to complete their execution.
Single Invocation Test run on a two-node active-active FortiSOARâ„¢ cluster
| Number of alerts created in FortiSOARâ„¢ | Total time (in seconds) taken to execute all the playbooks | Total number of playbooks executed in FortiSOARâ„¢ |
| 1 | 3.41 | 2 |
| 5 | 2.38 | 6 |
| 10 | 3.23 | 16 |
| 25 | 5.79 | 37 |
| 50 | 10.80 | 73 |
| 100 | 20.70 | 144 |
Note: Once this test is completed, refer to the pre-test conditions before starting a new test.
Test 3: Perform Ingestion in FortiSOARâ„¢ using the FortiSIEM Ingestion Playbook and after the alerts are created execute "Extraction" and "Enrichment" playbooks
Description of the Test
The test was executed using an automated testbed that starts FortiSIEM ingestion which in turn created alerts in FortiSOARâ„¢. Once the alerts are created in FortiSOARâ„¢, "Extraction" and "Enrichment" playbooks are triggered and the total time taken for all the extraction and enrichment playbooks to complete their execution is calculated.
Important: The setup for this test is exactly the same, however this test additionally requires the "VirusTotal" connector to be configured.
Steps followed
- Created the alerts using the FortiSIEM Ingestion playbook.
- Once the alerts are created, the "Extraction" playbooks are triggeredThe JSON for the sample playbooks that have been used are attached with this article (Test_2_Info_Json_Files.zip) so that you can run the same tests in your environment to see the performance in your version/hardware platforms. Or, if you want to do some additions that are specific to your environment, you can also tweak the existing playbooks.
The playbooks perform the following steps:- Declares variables using the "Set Variable Step".
- Updates the existing indicator list using mapping.
- Retrieves indicators from the source data of the alert.
- Creates indicators in the "Indicators" Module.
- Link alerts to the indicators.
- Update Alert State.
- Once the indicators were extracted, the "Enrichment" playbooks are triggered and they perform the following steps:
- Matches the IP in an internal subnet through the "Utilities" subnet.
- Validates whether the IP is Private or Public.
- Performs enrichment using the "Utilities" connector, if the IP is "Private".
- Performs enrichment using the "VirusTotal" connector, if the IP is "Public".
- Updates the indicator status based on the IP’s vulnerability.
- Updates the state of the indicator State.
Observations
The data in the following table outlines the number of alerts ingested, the total time taken to ingest those alerts, and the total time taken for all the triggered playbooks to complete their execution.
Single Invocation Test run on a two-node active-active FortiSOARâ„¢ cluster
| Number of alerts created in FortiSOARâ„¢ | Total time (in seconds) taken to execute all the playbooks | Total number of playbooks executed in FortiSOARâ„¢ |
| 1 | 5.39 | 4 |
| 5 | 6.46 | 16 |
| 10 | 8.59 | 31 |
| 25 | 19.44 | 76 |
| 50 | 35.66 | 151 |
| 100 | 1 minute 6 seconds | 301 |
Note: Once this test is completed, refer to the pre-test conditions before starting a new test. Also, note that enrichment of playbooks makes API calls over the Internet, and the times mentioned in this table to execute playbooks is inclusive of this time.
Sustained Invocation Test for the HA active-active cluster of two FortiSOARâ„¢ nodes
Description
A sustenance test was also conducted with the configuration as defined in "Test 2", i.e., the test is executed by manually triggering FortiSIEM Ingestion playbook that creates alerts in FortiSOARâ„¢. Once the alerts are created in FortiSOARâ„¢, an "Extraction" playbook is triggered and the total time taken for all the extraction playbooks to complete their execution is calculated.
Number of alerts: 100/min
Duration: 12 hours
Playbooks configured: As defined in Test 2 comprising of "Ingestion" and "Indicator Extraction" playbooks.
Total number of playbooks executed: 72720
Results
The system performed well under the sustained load. All 72000 alerts were successfully ingested and all the extraction playbooks were successfully completed without any queuing.
Graphs
The following graphs are plotted for the vital statistics for the HA cluster that was under test during the period of the test run.
Note: All the graphs included in this section are from the Primary/Active Node.
CPU Load Average Utilization Graph
Analysis of CPU load average utilization when the test run was in progress on the appliance:
Using the system resources specified in the "Environment" and "Pre-Test Conditions" sections, it was observed that while the "Sustenance Test" was running the CPU utilization was normal and the performance of the system did not get impacted.
Memory Utilization Graph
Analysis of memory utilization when the test run was in progress on the appliance:
Using the system resources specified in the "Environment" and "Pre-Test Conditions" sections, it was observed that while the "Sustenance Test" was running the Memory utilization was around 10GB.
Redis PB Queue Count Graph
Analysis of Redis PB Queue Count when the test run was in progress on the appliance:
Using the system resources specified in the "Environment" section and tunables configured as mentioned in the "Pre-Test Conditions" section, it was observed that while the "Sustenance Test" the redis_pb_queue was at 0. It went to 4 only once and again came back to 0. This means that no playbooks were getting queued and all the required playbooks associated with the alerts were getting completed in a minute.
IO Wait Graph
Analysis of IO Wait when the test run was in progress on the appliance:
Using the system resources specified in the "Environment" and "Pre-Test Conditions" sections, it was observed that while the "Sustenance Test" was running the IO Wait time was normal, with the average IO Wait being around 1% of the CPU idle time.
Read/Write IO Wait Graph for ElasticSearch
Analysis of Read/Write IO Wait for ElasticSearch when the test run was in progress on the appliance:
Using the system resources specified in the "Environment" and "Pre-Test Conditions" sections, it was observed that while the "Sustenance Test" was running the "Read" Wait for the ElasticSearch disk was almost 0 milliseconds. The "Write" Wait for the ElasticSearch disk averaged around 30 milliseconds, with the maximum wait of 40 milliseconds and the minimum wait of 1 millisecond.
Read/Write IO Wait Graph for PostgreSQL
Analysis of Read/Write IO Wait for PostgreSQL when the test run was in progress on the appliance:
Using the system resources specified in the "Environment" and "Pre-Test Conditions" sections, it was observed that while the "Sustenance Test" was running, the "Read" Wait for the PostgreSQL disk averaged around 0 millisecond. The "Write" Wait for the PostgreSQL disk averaged around 8 milliseconds, with the maximum wait of 25 milliseconds and the minimum wait of 1 millisecond.
