Technical Note: CSTN 00038 - CyOPs Performance Benchmarking for v5.0.0

Overview

This document details the performance benchmark tests conducted in CyberSponse labs. The performance benchmarking tests were performed on CyOPs™ version 5.0.0 Build 866.

Objective

The objective of this performance test is to measure the time taken to create alerts in CyOPs™, and complete the execution of corresponding playbooks on the created alerts on a single-node CyOPs™ appliance and a cluster setup of CyOPs™.

The data from this benchmark test can help you in determining your scaling requirements for CyOPs™ instance to handle the expected workload in your environment.

Environment

CyOPs™ Virtual Appliance Specifications

Operating System Specifications

Pre-test Conditions

At the start of each test run -

• The test environment contained zero alerts.
• The test environment contained only the CyOPs™ built-in connectors such as IMAP, Utilities, etc.
• The system playbooks were deactivated and there were no running playbooks.
• The playbook execution logs were purged.

Details of the CyOPs™ Performance Benchmarking Test

The test was executed using an automated test bed that initiated HTTPS calls per clock tick ( x alerts ingested per second ) which created alerts in CyOPs™ and then triggered a playbook for each alert created. Steps are as follows:

1. The alerts were created using JMeter to simulate parallel invocation of the API - ‘/api/3/alerts/’.
2. When an alert was created, a post-create playbook is triggered which performs the following steps: 
   • Declare variables using the Set Variable Step
   • Extract artifacts from the source data of the alert using the “CyOPs: Extract Artifacts from String” action of the CyOPs Utilities connector.
   • Add the extracted artifacts in the “Closure Notes” field of the alert.
   • Update the status of the alert to “Closed”.

Observations

The data in the following tables outlines the number of alerts ingested in a clock tick, the total time taken to ingest those alerts, and the total time taken for all the playbooks triggered to finish execution.

Single Invocation Test run on a single-node CyOPs™ appliance 

Single Invocation Test run on a two-node Active-Active CyOPs™ cluster 

Sustained Invocation Test

In the sustenance test conducted on a two-node Active-Active CyOPs™ cluster, we could ingest 100 Alerts every 30 secs over 24 hours and observed that 176904 alerts were generated and corresponding playbooks successfully completed.

In the sustenance test conducted on a single node machine we could ingest 100 Alerts every 30 secs over 24 hours and observed that 15017 alerts were generated and corresponding playbooks successfully completed.

Conclusion

Based on this test, we conclude, that CyOPs™ could process an average of 6250 alerts in an hour in a single node and 7084 alerts in an hour in a two-node Active-Active CyOPs™ cluster. This includes creation of alerts, and running corresponding playbooks to process the alerts.

Notes

In a production environment the following factors might vary, which could affect the observations:

1. The size of alert data.
2. The number of playbooks that are being executed in parallel for each alert (ex, system playbook for notification or triage/investigate playbooks).
3. The number of steps in each playbook.
4. The network bandwidth especially for outbound connections to applications such as VirusTotal.