Technical Tip: CSTN 00043 - CyOPs Performance Benchmarking for v5.0.1
This article describes the performance benchmark tests conducted in CyberSponse labs.
The performance benchmarking tests were performed on CyOPs™ version 5.0.1 Build 098.
Solution
Objective.
The objective of this performance test is to measure the time taken to create alerts in CyOPs™, and complete the execution of corresponding playbooks on the created alerts on a single-node CyOPs™ appliance and a cluster setup of CyOPs™.
The data from this benchmark test can help you in determining your scaling requirements for CyOPs™ instance to handle the expected workload in your environment.
Environmnent.
CyOPs™ Virtual Appliance Specifications
| Component | Specifications |
| CPU | 8 CPUs |
| Memory | 16 GB |
| Storage | 250 GB virtual disk running on top of Samsung SSD 360 Pro model attached to VMware ESX server |
Operating System Specifications
| Operating System | Kernel Version |
| CentOS 7 | 3.10.0-957.5.1.eI7.x86_6 |
Pre-test Conditions
At the start of each test run -
- The test environment contained zero alerts.
- The test environment contained only the CyOPs™ built-in connectors such as IMAP, Utilities, etc.
- The system playbooks were deactivated and there were no running playbooks.
- The playbook execution logs were purged.
Details of the CyOPs™ Performance Benchmarking Test
The test was executed using an automated testbed that initiated HTTPS calls per clock tick (X alert API calls triggered at once) which created alerts in CyOPs™ and then triggered a playbook for each alert created. Steps are as follows:
- The alerts were created using JMeter to simulate parallel invocation of the API - ‘/api/3/alerts/’.
- When an alert was created, a post-create playbook is triggered which performs the following steps:
- Declare variables using the Set Variable Step
- Extract artifacts from the source data of the alert using the “CyOPs: Extract Artifacts from String” action of the CyOPs Utilities connector.
- Add the extracted artifacts in the “Closure Notes” field of the alert.
- Update the status of the alert to “Closed”.
Setup
The test is invoked in two different setups:
- A single node CyOPs™ machine : This machine is standalone machine for CyOPs™ and all calls are directly ingested by this machine.
- A High Availability (HA) two-node cluster with an HA Proxy: This setup is a cluster of two CyOPs™ machines which are joined in the Active-Active state using the CyOPs™ HA feature. This setup also contains an HA proxy, which acts as a load balancer. All the calls are directed to the HA proxy load balancer, which functions in a round-robin method to distribute the load of calls.
Observations
The data in the following tables outlines the number of alerts ingested in a clock tick, the total time taken to ingest those alerts, and the total time taken for all the playbooks triggered to finish execution.
Single Invocation Test run on a single-node CyOPs™ appliance
Number of alerts created in CyOPs™ | Total time taken to create all alerts in CyOPs™ (in seconds) | Total time taken to execute all Playbooks (in seconds) |
25 | 2 | 10.603 |
50 | 4 | 22.578 |
100 | 9 | 45.108 |
Single Invocation Test run on a two-node Active-Active CyOPs™ cluster
Number of alerts created in CyOPs™ | Total time taken to create all alerts in CyOPs™ (in seconds) | Total time taken to execute all Playbooks (in seconds) |
25 | 3 | 4.670 |
50 | 3 | 5.704 |
100 | 7 | 14.616 |
Sustained Invocation Test.
In the sustenance test conducted on a two-node Active-Active CyOPs™ cluster, we could ingest 200 Alerts every 20 secs over 12 hours and observed that 202788 alerts were generated and corresponding playbooks successfully completed.
In the sustenance test conducted on a single node machine we could ingest 100 Alerts every 20 secs over 12 hours and observed that 117592 alerts were generated and corresponding playbooks successfully completed.
* The number of alerts ingested in the system are the same as the alerts generated by the performance tool.
Conclusion.
Based on this test, we conclude that CyOPs™ could process an average of 9799 alerts in an hour in a single node and 16788 alerts in an hour in a two-node Active-Active CyOPs™ cluster. This includes creation of alerts, and running corresponding playbooks to process the alerts.
Notes.
In a production environment the following factors might vary, which could affect the observations:
