ITOps v1.3-Performance Test Report

Overview

SmartOps ITOps manages alerts and notifications from multiple monitoring agents for an organization.

It supports automatic creation of Service Now tickets against alerts and provides intelligent root-cause suggestions by correlating with existing open tickets.

It avoids creation of duplicate tickets by identifying similar open tickets or root tickets based on the dependency topology.

Performance Testing – Workload Parameters

The following parameter define the workload for executing the performance testing of ITOps for Queue channel during multiple test executions.

 

Test Environment - Infrastructure setup

ITOps components, IHub, Clones, Databases & Message Broker are hosted in Kubernetes environment in Azure cloud.

Performance Testing is conducted with the following configuration of Kubernetes Cluster hosted in Azure. The Kubernetes ITOps cluster comprises of 2 different types of node pools such as Application Node Pool, Persistent Node Pool.

The below table summarizes the hardware configuration & Number of VM & components hosted in each Node Pools.

 

 

Performance test environment setup

To replicate a production-based Infrastructure environment IHub Lite & SolarWinds based mock service are hosted in an Azure VM hosted at Azure cloud in a different virtual network.

IHub main along with ITOps components are hosted at Kubernetes environment in Azure cloud in another virtual network.

 

 

Performance Testing - Software Used

Following Tools used as part of performance testing

 

Performance Testing – ITOps components & Replicas

The below table provides details of docker containers of different components associated with ITOps from, Clones, ITOps, Clones, iHub, Database & Message Broker.

This also provides detail of components identified for Autoscaling and the criteria defined for Autoscaling. As part of ensuring availability of non-scalable components to avoid failures of any components during document processing, 2 instances of each components are by default available in the Kubernetes cluster.

 

 

Following are the details of replicas configured for Database & Message Broker in Kubernetes cluster MySQL is deployed as a Managed service at Azure.

 

 

Autoscaling Configurations

Following are the CPU threshold limits, Replicas (Minimum & Maximum) configured for each component identified for Autoscaling in Kubernetes cluster for ITOps.

 

Performance Test Approach

Performance Testing of ITOps is conducted by sending system alerts & notifications captured by SolarWinds monitoring agent. A SolarWinds monitoring agent based mock service is created to simulate different types of System alerts such as UP, DOWN, Warning, Critical alerts & notifications alerts are simulated by the mock service.

This service is polled continuously by IHub Lite API scanner channel in a fixed interval and the polled data is pushed to ITOps Queue system from which the alerts are processed further by IHub main & ITOps. SolarWinds based mock service simulates alerts archived from a production-based system.

Test is conducted by simulating different counts of system alerts by mock service in a specific interval at different alert times. This helps in creating system alerts workload like a production-based environment

This test helps us to identify the throughput of alerts that is successfully processed by ITOps and the time taken in correlation of alerts based on the improvements introduced by platform team in this release to reduce the alert correlation time compared with previous version of ITOps.

 

Performance Test -Summary

Performance Testing was conducted on ITOps by sending SolarWinds based alerts samples continuously for a long duration which is simulated by SolarWinds mock service.

Multiple long duration test execution has been conducted with different counts of alerts simulated every minute by SolarWinds mock service like a production-based environment. The metrics captured & Observations/findings from the test execution helps in identifying the improvements needed in alert processing by different ITOps components.

IHub lite API scanner channel polls the mock service continuously in an interval of 1 minute and IHub lite channel publishes the alerts to ITOps Queue from which further processing of alerts starts.

In order to handle the increased workload of alerts Autoscaling is enabled for clones engine, itops correlation & alert mapping containers. Autoscaling is enabled based on the CPU usage threshold of 80% with maximum replicas of 4.

This test report is prepared based on the metrics collected from the last two long duration test execution conducted using test data archived from a production system.

Following are the counts of alerts received at IHub Lite, IHub Queue channel & Alert store. There was no missing of alerts observed in the test execution.

 

Alert Correlation Time

In order To replicate the alert simulation pattern in a production system, The test is conducted by simulating different counts of alerts from SolarWinds mock service every minute, this type of varying alert simulation pattern ensures that each time the SolarWinds mock service is polled, the count of alerts received will be different which is similar to a production system. This also helps to capture the metrics on alert correlation with varying count of alerts reaching ITOps.

Following are the threshold count of alerts set for correlation workflow, median count of alerts received per minute from SolarWinds mock service and the duration of the schedulers for Correlation, Ticket status Update & Auto closure flap cluster.

Alert Threshold count per execution of workflow represents the count of alerts taken by correlation workflow during each execution of the workflow as part of alert correlation.

 

 

Following are the time taken for alert correlation during the 2 test executions conducted with above mentioned configuration of the count of alerts and workflow scheduler time.

Test 1

Time taken for correlation of each alerts is 3.38 seconds (p90) and median time is 1.76 seconds

 

Test 2

Time taken for correlation of each alerts is 3.73 seconds (p90) and median time is 1.78 seconds

 

Following is the graphical representaion of time taken for correation of each alerts.

Test 1

Test 2

IHub Alert Processing Time

The following graph shows the time taken in the processing of alerts in IHub Queue channel. This also involves the time taken in the transformation of alerts received from IHub Lite channel to SmartOps Platform message format.

Alerts are received as batches each time when iHub polls the ITOps Queue with each batch having different counts of alerts simulated from SolarWinds mock service.

Median time taken by iHub in processing each alert received in different batch is 0.08 seconds

Test 1

Test 2

Median time taken by iHub in processing each alert received in different batch is 0.14 seconds

 

Performance Metrics Captured

Volume & type of Alerts sent

This captures total count of alert samples and count of major type of types of system alerts & notifications received at IHub Queue channel.

Clones Workflow Execution Time

This captures the time taken (Queue & Execution time) for the execution of workflows such as Correlation, Ticket update Scheduler & Auto close flap cluster by clones engine as part of processing different system alerts.

Each workflow has different steps of micro actions and there is a Queue phase & Execution phase involved in each of steps.

Below table summarizes the Average time taken in Queue & Execution phase involved in each step associated with a workflow during the test execution.

 

 

Performance Test – Infrastructure Resource Usage

Following are the CPU & Memory usage of various components in ITOps during the test executions.

Application components

Test 1

Test 2

 

Following are the percentage wise use of CPU & Memory of each application components,

Max usage represents the maximum value reached in resource usage (CPU, Memory) during the test run.

POD resource limit represents the max limit set in each component Kubernetes POD.

 

Test 1

Test 2

 

Test 1

Test 2

Test 1

 

Test 2

 

 

Test 1

Test 2

 

Test 1

Test 2

 

Test 1

 

Test 2

Databases & Message Broker

Test 1

Test 2

Following are the percentage wise use of CPU & Memory of each Database & Message Broker components of ITOps.

 

Test 1

 

Test 2

Test 1

Test 2

 

Test 1

 

Test 2

 

POD Autoscaling metrics

Following are the usage of replicas of POD’s of the ITOps components identified for Auto scaling. During both test executions its observed that apart from clones engine none of the components which had autoscaling enabled had any change on the pod replica count from its minimum value.

The below table summarizes the average, Max POD replica usage of the following ITOps components identified for autoscaling.

Test 1

 

Test 2

 

Following graph shows the clones engine POD replica usage during the test executions.

Test 1

Test 2

‎

Performance test Observations & Recommendations

 

• Based on the two test executions conducted with 21090, 21250 alerts spanned for a duration of 23 hrs & 25 hrs, its observed that there is significant improvement brought in on time taken for alert correlation in ITOps 1.3 compared to the previous versions of ITOps.

• In first test execution time taken by each alert for correlation is 3.38 seconds (p90) and median time is 1.76 seconds and in second test execution the alert corelation time is 3.73 seconds (p90) & median time is 1.78 seconds per alert whereas in ITOps 7.1.2 it was 17.62 seconds (p90).

• Both Test execution of ITOps 1.3 was conducted with each execution of correlation workflow having 55 alerts for alert correlation. With throughput of 55 alerts for correlation it was able to process each alert in 3.38 seconds (P90) & 3.73 seconds (p90). This is a good performance improvement as the ITOps system was able to process all the alerts received without any failures in reduced processing time during correlation of alerts.

• The introduction of an API for handling Alert Correlation in ITOps 1.3 had considerably reduced the workflow execution time of Correlation Workflow. The average execution time of correlation workflow in ITOps 1.3 is 1.51 seconds compared with 294.57 seconds in ITOps 7.1.2.

• Its observed that resource usage of alert correlation is relatively very less compared to resource limit set in both versions of ITOps. The CPU usage in both versions is only 0.001 core whereas the resource limit set is 1 core, with the high volume of alerts used in performance test only 0.001 core was needed, so the CPU resource limit can be reduced to a lower value.

• Similar to CPU usage, in alert mapping component memory usage is also relatively very less compared with resource limit set (1.5 GIB), In both version performance test the memory usage is below 100 MiB, so the resource limit of memory can also be reduced to a lower value.

• Its observed that memory usage of Elastic Search is reaching very close to resource limit value in all the 3 POD’s, In ITOps 7.1.2 resource limit of memory was 4 GiB and in ITOps 1.3 it was increased to 6 GiB, still the memory usage is very close to the resource limit value. So, platform team can have any analysis on the high memory usage.

• *** Based on the resource usage collected from the two test executions conducted its observed that resource usage (CPU & Memory) of alert correlation, alert mapping, clones sense queue, iHub component, iHub Service is very less compared with respect to POD resource limit being set.

• To identify the ideal resource configurations of those components another test execution will be conducted by setting lower resource values in CPU & Memory. The test result of that execution will be shared based on its completion.

Performance test metrics comparison- ITOps 1.3 & 7.1.2

Time taken for Alert correlation

 

There is significant improvement introduced in time taken for alert correlation in ITOps 1.3, it took only 3.38 seconds (P90) for each alert correlation where as in ITOps 7.1.2 it was 17.62 seconds (P90).

Workflow Execution Time

In correlation workflow execution time also there is considerable improvement in ITOps 1.3 compared with 7.1.2, the average workflow execution time is only 1.51 seconds where as in 7.1.2 it was 294.60 seconds

 

 

 

Resource Usage

Application Components

ITOps 1.3

ITOps 7.1.2

 

Database & Message Broker

ITOps 1.3

 

ITOps 7.1.2

 

Infrastructure Cost

Following is the approximate pricing for the infrastructure used for the performance test of ITOps 1.3.

More details on the Hardware Infrastructure used for Performance test is shown below.

 

 

Please note that this is indicative and actual billing could vary based on network usage, operations, data etc.

 

ITOps Infrastructure configuration

Performance test Execution results

Raw data collected as part of test execution is available on the following SharePoint location

Test Data