Contents
|
Acronym / Item |
Expansion / Definition |
|
Metrics |
Measurements obtained from executing the performance tests and represented on a commonly understood scale. |
|
Throughput |
Number of requests handled per unit of time. Example requests/second. |
|
Workload |
Stimulus applied to the system to simulate actual usage pattern regarding concurrency, data volumes, transaction mix. |
|
PWF |
Package Workflow |
|
POD |
Pod is a Kubernetes abstraction that represents a group of one or more application containers (such as Docker), and some shared resources for those containers. |
|
DU |
Document Understanding, a SmartOps proprietary AI based solution for Invoice extraction. |
Performance test was conducted for SmartOps Kubernetes based Infrastructure hosted in Azure Cloud. Test environment was deployed with Smart Extract pipeline and its related containers only. Vespa pipeline and its related containers are excluded in this test executions.
The below table summarizes the hardware Infrastructure along with Node count used for the Smart Extract infrastructure.
|
Kubernetes Nodes |
Hardware Infrastructure |
Node Count |
|
|
Min |
Max |
||
|
Application Node Pool Components DU (Smart-extract, smart-extract-predict, rest, pipeline, scheduler, invoice-split, page-split , invoice-image-ocr), Clones Engine, pwf-invoice-extraction-scheduler, pwfinvoice-extraction-listener, pwf-invoice-extraction-api, ie-ui |
Azure D8sv3 CPU - 8 vCPU Core RAM: 32 GB |
4 |
6 |
|
Persistent Pool Components Mongo, MinIO, RabbitMQ, Elastic Search, Kibana |
Azure D4sv3 CPU - 4 vCPU Core RAM: 16 GB |
3 |
6 |
|
MySQL |
Azure Managed MYSQL General Purpose, 2 Core(s), 8GB RAM, 100 GB (Auto Grow Enabled) |
NA |
NA |
Following are the details of replicas configured for Database & Message Broker in Kubernetes cluster MySQL is deployed as a Managed service at Azure.
|
Stack Name |
Container Name |
No of instances |
|
|
Database |
mongo |
3 |
|
|
minio |
2 |
||
|
elasticsearch |
3 |
||
|
mysql |
Azure Managed Service |
||
|
Message Broker |
rabbitmq |
3 |
|
Following are the CPU threshold limits, Replicas (Minimum & Maximum) configured for each component identified for Autoscaling in Kubernetes cluster for FinOps.
|
Container name |
CPU Threshold |
min replicas |
max replicas |
|
|
Clones-engine |
80% |
2 |
4 |
|
|
du-pipeline |
80% |
2 |
4 |
|
|
du-rest |
80% |
2 |
4 |
|
Based on the optimal Invoice processing observed in previous tests, Smart Extract Engine and Invoice Image OCR POD replicas are set to 3. All the tests are conducted with this configuration.
|
Smart Extract Engine |
Invoice Image OCR |
|
|
3 |
3 |
|
The below table provides details of docker containers of different components associated with FinOps from Document Understanding (DU), Clones, PWF, Database & Message Broker.
This also provides detail of components identified for Autoscaling and the criteria defined for Autoscaling. As part of ensuring availability of non-scalable components to avoid failures of any components during document processing, 2 instances of each components are by default available in the Kubernetes cluster.
|
Stack Name |
Container Name |
Autoscaling Enabled |
Scaling Criteria |
|
Document Understanding |
du-rest |
Y |
Based on CPU Usage |
|
du-scheduler |
N |
|
|
|
du-pipeline |
Y |
Based on CPU Usage |
|
|
du-invoice-split |
N |
|
|
|
du-tilt-correct |
N |
|
|
|
du-smart-extract |
N |
|
|
|
du-invoice-image-ocr |
N |
|
|
|
Clones |
clones-sense-queue |
N |
|
|
clones-engine |
Y |
Based on CPU Usage |
|
|
PWF |
pwf-Invoice-extractionlistener-du |
N |
|
|
pwf-invoice-extractionlistener |
N |
|
|
|
pwf-invoice-extraction-api |
N |
|
|
|
pwf-invoice-extractionscheduler |
N |
|
DU Project configuration settings
|
DU Project setting |
Value |
|
Engine Pipeline |
Smart Extract |
|
Language Classifier |
Disabled |
|
Feedback Learning (HITL) |
Disabled |
|
Preprocessors Enabled |
Barcode Page Split, Skip Dup Validation |
Following Tools used as part of performance testing
|
Tool |
Version |
Description |
|
JMeter |
5.1.1 |
Implementing Performance testing |
|
Prometheus |
|
Capture resource utilization on server side. |
|
Grafana |
|
Dashboard to view resource utilization. |
|
Microsoft Excel |
|
Analyzing test results &reports. |
|
SmartVision |
2.4 |
SmartOps SmartVision application. |
Three test items are identified in this release. Test data used and the approach for each test is detailed in each section below.
Test is executed using Invoice samples of 51 per batch zip, each Invoice has 2 pages the first page being the Invoice scan page and second page is the invoice page. Different vendor-based sample invoices are used consisting of Invoices of different complexity categorized as low, medium, and high.
|
Batch file parameter |
Value |
|
Pages per Invoice document. |
2 |
|
No of Invoices per batch in a zip file |
51 (102 pages) |
|
Invoice Batch |
3 different variants of Invoices of High, Medium, and low complexity |
|
Number of fields extracted per invoice |
22 |
|
Document Format |
|
|
Zip file contents (Combined, Individual). |
Combined PDF of Invoices |
|
Insight Fields |
14 |
|
Line Items |
8 |
Conduct a test using this invoice. Repeat the test for 3 iteration to verify the consistent result. Capture the test metrics and compare the result with the 7.2 test results.
Single PDF invoices are used for this test and the configuration is as below. Each batch is a zip of 10 individual PDF files. Sample invoices from customer is used here.
|
Batch file parameter |
Org 1 |
Org 2 |
|
File type |
|
|
|
Invoice Batch |
Zip |
Zip |
|
Number of fields extracted per invoice |
22 |
22 |
|
Document Format |
|
|
|
File contents |
Individual PDF inside zip |
Individual PDF inside zip |
|
Insight Fields |
14 |
14 |
|
Line Items |
8 |
8 |
Create two organization and create one set of projects in pwf and DU. Schedule a batch of (10 invoices) invoice doc with a time interval of 5 minutes in both pwf projects. Update the FTP with other batch file every 5 minutes interval. A total of 15 such invoice batches are configured to execute from each ORG. A total of 150 invoices from each ORG (total of 300 in the system). Run the test until completion of files processing in both the organizations. Collect the test result and analyze the system performance.
Repeat the same test with different time interval of 10 mins.
This is like 2 ORG test. In this test, we will again consider more customized setting in the DU Project configuration like ‘locale’ which decides the model to be considered in processing the invoice documents. ’US’ and ‘AUS’ locale will be considered as two models in each ORG.
This test is executed after the 2 ORG Test is completed. Invoices processed in 2 ORG is used for navigating UI actions to capture the UI/API time for navigation of different user actions. Below user actions are identified for this test and will be executed for a single UI user with a background load of 4 users accessing the server.
Insights page loading of pdf and insight data.
Navigate to next/previous invoice in the batch.
Save & Approve scenario.
Reject/Escalate scenario.
Point & Select
Performance test servers are hosted in east US region. Separate tests are executed from VM and from local (Kochi, South India). As the VM is hosted in the same region as the application server resides, there won’t be any delay for data transportation and switching of regions. Another test is conducted from local which is a different region than the servers.
Average time taken to process 51 invoices is 18.75 minutes.
|
Test Execution |
DU Time Taken (Minutes) |
|
|
First file processed |
Last file processed |
|
|
Test 1 |
2.93 |
18.98 |
|
Test 2 |
2.87 |
18.50 |
|
Test 3 |
2.82 |
18.77 |
Comparison between 2.3 and 2.4. The avg time taken to process 51 invoices is improved by 4.13 minutes in 2.4 compared to 2.3.
|
Test
|
DU Time Taken (Minutes) |
|
||
|
2.3 |
|
2.4 |
||
|
51 invoices |
22.88 |
|
18.75 |
|
Two organizations ORG1 and ORG2 are configured with different DU and PWF projects.
|
ORG |
Invoice Fields Locale |
Test schedule |
|
ORG1 |
US |
15 batches with 10 invoices in each batch. |
|
ORG2 |
AUS |
15 batches with 10 invoices in each batch. |
As both the projects are configured with different Locale, the ‘se_model_id’ will be different for them. The test execution result of this model is as below.
Throughput is improved in 2.4 to 137 (invoice / hour) compared to 123 in 2.3.
|
Test Name |
Batch size |
Inv / ORG |
Failure |
Total files |
Test Duration |
Effective time (secs) |
Throughput (Per Hour) |
|
300 inv_5 min |
10 |
150 |
1 |
300 |
7828 |
7828 |
137.05 |
|
10 |
150 |
1 |
7782 |
UI time is captured using a stopwatch manually from the chrome browser during the 10 users load executed from the JMeter tool.
PDF view UI time taken from local machine during the 10 users load on the server.
|
|
|
Time in seconds |
|
|||
|
No Throttling (Disabled) |
Fast 3G |
Slow 3G |
||||
|
User Action |
End-to-end UI Time |
API time |
End-to-end UI Time |
API time |
End-to-end UI Time |
API time |
|
PDF view - sample 1 |
3.62 |
1.85 |
4.61 |
1.95 |
19.09 |
16.43 |
|
PDF view - sample 2 |
4.06 |
2.19 |
3.71 |
2.1 |
155.26 |
66 |
|
PDF view - sample 3 |
3.07 |
1.85 |
3.36 |
1.71 |
6.7 |
4.11 |
|
PDF view - sample 4 |
3.61 |
2.06 |
3.86 |
3.16 |
6.57 |
4.1 |
|
PDF view - sample 5 |
5.04 |
2.73 |
4.45 |
3.32 |
8.45 |
4.04 |
PDF view UI time taken from virtual machine (VM) during the 10 users load on the server.
|
|
|
Time in seconds |
|
|||
|
No throttling (Disabled) |
Fast 3G |
Slow 3G |
||||
|
User Action |
End-to-end UI Time |
API time |
End-to-end UI Time |
API time |
End-to-end UI Time |
API time |
|
PDF view - sample 1 |
2.76 |
0.92 |
2.96 |
1.17 |
9.97 |
4.27 |
|
PDF view - sample 2 |
1.9 |
0.842 |
2.81 |
1.18 |
6.73 |
4.12 |
|
PDF view - sample 3 |
2.05 |
0.927 |
3.02 |
1.15 |
6.44 |
4.11 |
|
PDF view - sample 4 |
2.35 |
0.971 |
5.15 |
1.18 |
6.93 |
4.17 |
|
PDF view - sample 5 |
2.75 |
1.45 |
7.19 |
1.53 |
6.62 |
4.16 |
The following graph represents the time-based execution of the different steps of DU as part of the processing of the 51 Invoice sample for one of the test executions.
This is the time taken to execute the following workflow by clones engine as part of document processing.
Invoice_PWF_PushToDUSched is a scheduled workflow which polls the FTP folder to fetch the Invoice batch zip file and provides the zip file to the corresponding Document Understanding Project for data extraction.
Invoice_PWF_CheckRoleCondition workflow validates the role-based permission needed as part of the processing of invoice documents.
The below table summarizes the average execution time taken by the clones workflows.
|
|
|
Clones Workflow (Seconds) |
|
|
|
Invoice Batch |
Executions |
CheckRoleCondition |
Invoice PWF Installation |
PushToDUSched |
|
51 invoices |
Test1 |
16.09 |
3.12 |
161.42 |
|
Test2 |
11.07 |
3.09 |
41.82 |
|
|
Test3 |
17.65 |
2.75 |
53.96 |
|
There are several steps involved in the processing of an Invoice document by Documents Understanding application. Time taken for the execution of different steps in DU as part of Invoice extraction.
|
DU Steps |
|
Execution Time (Seconds) |
|
||
|
AVG |
MIN |
MAX |
Occurrence per batch |
Extended Time per batch |
|
|
INITIALIZE |
0.28 |
0.11 |
0.80 |
155 |
43.37 |
|
VALIDATE |
0.12 |
0.05 |
0.48 |
155 |
18.65 |
|
ZIP_EXTRACT |
1.58 |
1.58 |
1.58 |
1 |
1.58 |
|
INV_DOC_SPLIT |
384.64 |
384.64 |
384.64 |
1 |
384.64 |
|
INV_PAGE_SPLIT |
0.44 |
0.24 |
0.82 |
51 |
22.51 |
|
INV_IMAGE_OCR |
14.30 |
3.13 |
74.12 |
102 |
1458.13 |
|
SMART_EXTRACT_PREPROCESSOR |
4.31 |
1.90 |
10.64 |
102 |
439.68 |
|
SMART_EXTRACT |
11.91 |
9.50 |
20.66 |
102 |
1214.32 |
|
SMART_EXTRACT_POSTPROCESSOR |
2.35 |
0.40 |
6.26 |
102 |
239.36 |
|
SMART_EXTRACT_FOI |
14.99 |
1.33 |
39.21 |
102 |
1529.34 |
|
TRIGGER_PARALLEL_VESPA_SE |
2.14 |
1.19 |
4.09 |
102 |
218.48 |
|
INV_RESULT_AGGREGATE |
0.18 |
0.08 |
0.47 |
51 |
9.37 |
|
FINALIZE |
1.09 |
0.25 |
3.95 |
359 |
391.43 |
|
PDF_SEARCH |
5.69 |
0.81 |
41.92 |
102 |
580.82 |
Each steps of Document extraction in DU, there is a Queue phase associated. The below table summarizes the time spent in the Queue phase of each of the DU steps.
|
DU Steps |
Queue Time (Seconds) |
|
|||
|
AVG |
MIN |
MAX |
Occurrence per batch |
Extended Time per batch |
|
|
INITIALIZE |
0.72 |
0.31 |
2.66 |
155 |
111.99 |
|
VALIDATE |
0.52 |
0.24 |
1.51 |
155 |
80.76 |
|
ZIP_EXTRACT |
0.32 |
0.32 |
0.32 |
1 |
0.32 |
|
INV_DOC_SPLIT |
0.49 |
0.49 |
0.49 |
1 |
0.49 |
|
INV_PAGE_SPLIT |
492.83 |
104.02 |
757.99 |
51 |
25134.41 |
|
INV_IMAGE_OCR |
83.40 |
0.32 |
173.37 |
102 |
8506.55 |
|
SMART_EXTRACT_PREPROCESSOR |
150.32 |
0.55 |
281.67 |
102 |
15332.37 |
|
SMART_EXTRACT |
5.27 |
0.27 |
24.47 |
102 |
537.10 |
|
SMART_EXTRACT_POSTPROCESSOR |
41.91 |
0.24 |
94.12 |
102 |
4274.86 |
|
SMART_EXTRACT_FOI |
123.65 |
0.33 |
234.32 |
102 |
12612.46 |
|
TRIGGER_PARALLEL_VESPA_SE |
0.45 |
0.22 |
1.62 |
102 |
45.72 |
|
INV_RESULT_AGGREGATE |
0.47 |
0.29 |
0.93 |
51 |
23.87 |
|
FINALIZE |
0.53 |
0.22 |
6.20 |
359 |
189.55 |
|
PDF_SEARCH |
2.31 |
0.27 |
30.11 |
102 |
236.09 |
A Queue phase is part of each DU steps when the number of messages reaches beyond the capacity of the individual step that can be processed. These additional messages are stored in the RabbitMQ message system.
During the test execution, this RabbitMQ Queue message metrics are captured in identifying in which Queue the de-Queuing process is slow. This will help in fine tuning of the DU component and there by improve the overall application performance.
The below graph represents the count of messages in different queue used by each DU steps during test execution.
90th Percentile (P90) 10 Users API execution time comparison between Local machine and VM.
|
Label |
90th Percentile API Time (P90) milli seconds |
||
|
Local Machine |
Virtual Machine (VM) |
||
|
/lock/11607/renew |
3211 |
139 |
|
|
/pages/image |
4183 |
596 |
|
|
alerts |
5556 |
108 |
|
|
correction/invoice_Save N Approve |
1771 |
783 |
|
|
du_projects |
7139 |
223 |
|
|
fetch-details |
3532 |
117 |
|
|
fetch-projects-for-a-user |
5569 |
770 |
|
|
get_summary |
5134 |
1588 |
|
|
get_summary_PROJECTID |
4344 |
1545 |
|
|
getdufields |
6356 |
351 |
|
|
get-run |
3197 |
282 |
|
|
lock-off |
2729 |
164 |
|
|
lock-on |
2345 |
195 |
|
|
lock-renew |
2644 |
169 |
|
|
pages/image |
3653 |
606 |
|
|
pages-image_INVOICEID |
1982 |
521 |
|
|
pending |
3258 |
1407 |
|
|
pwf/fetch-details |
3096 |
113 |
|
|
rebound-tables |
709 |
128 |
|
|
results_INVOICEID |
5338 |
750 |
|
|
run_get |
4121 |
1561 |
|
Below table summarizes the UI time captured using a stopwatch while navigating the test scenarios from the chrome bowser during the 10 users load executed from the JMeter tool.
|
UI user action time captured using a stopwatch (seconds) |
|||
|
User action from the browser |
Local machine |
VM |
|
|
PDF view - sample 1 |
4.34 |
3.04 |
|
|
PDF view - sample 2 |
3.9 |
2.8 |
|
|
PDF view - sample 3 |
3.37 |
4.25 |
|
|
PDF view - sample 4 |
3.26 |
5.18 |
|
|
PDF view - sample 5 |
4.63 |
2.76 |
|
|
next - sample 1 |
2.81 |
2.14 |
|
|
next - sample 2 |
2.44 |
2.76 |
|
|
next - sample 3 |
1.85 |
1.21 |
|
|
prev - sample 1 |
1.23 |
1.47 |
|
|
prev - sample 2 |
1.33 |
1.95 |
|
|
prev - sample 3 |
1.85 |
1.38 |
|
|
save & approve |
5.22 |
2.72 |
|
|
Escalate |
2.92 |
2.91 |
|
|
Reject |
2.88 |
2.3 |
|
|
Select draw pane |
1.77 |
1.79 |
|
|
Select point & select |
4.24 |
3.01 |
|
|
User action from the browser |
Local machine (2.3) |
Local machine (2.4) |
VM (2.3) |
VM (2.4) |
|
PDF view - sample 1 |
3.69 |
4.34 |
1.26 |
3.04 |
|
PDF view - sample 2 |
3.27 |
3.9 |
1.26 |
2.8 |
|
PDF view - sample 3 |
2.67 |
3.37 |
1.7 |
4.25 |
|
PDF view - sample 4 |
2.26 |
3.26 |
1.32 |
5.18 |
|
PDF view - sample 5 |
1.68 |
4.63 |
1.88 |
2.76 |
|
next - sample 1 |
2.86 |
2.81 |
1.64 |
2.14 |
|
next - sample 2 |
1.32 |
2.44 |
1.37 |
2.76 |
|
next - sample 3 |
2.31 |
1.85 |
1.49 |
1.21 |
|
prev - sample 1 |
1.87 |
1.23 |
1.32 |
1.47 |
|
prev - sample 2 |
3.25 |
1.33 |
1.81 |
1.95 |
|
prev - sample 3 |
1.93 |
1.85 |
2.1 |
1.38 |
|
save & approve |
2.96 |
5.22 |
2.01 |
2.72 |
|
escalate |
2.69 |
2.92 |
1.92 |
2.91 |
|
Reject |
2.09 |
2.88 |
1.62 |
2.3 |
|
Select draw pane |
2.67 |
1.77 |
1.65 |
1.79 |
|
Select point & select |
3.25 |
4.24 |
1.99 |
3.01 |
As part of ensuring high availability, 2 PODS of each components are deployed in Kubernetes. The data represented below are the maximum CPU cores & memory usage of both the PODs.
Test execution raw data is available on the following SharePoint location - 2.4 raw data