SmartOps v7.0 Test Summary Report
Click here to see this page in full context
  1. Navigate To: SmartOps Document Center >
  2. SmartOps Document Center >
  3. Release Documentation >
  4. Platform >
  5. 7.0 >
  6. Test Summary Report

 

SmartOps v7.0 Test Summary Report

FinopsFinops

Executive Summary

End to end functional and regression testing of  Invoice Extraction PWF is completed.

Period of Testing: 05 th Oct 2020 to 17th Nov 202

Test Execution Summary

Sl No.

Area

Execution Status

Open Defects if any

1

Invoice Extraction PWF

Completed


JIRA Defects Tested 


FinOps

Jira ID

Test Status

Comments

SDI-181

Verified – QA Passed

 SDI-256

Verified – QA Passed

SDI-243

Verified – QA Passed

SDI-290

Verified – QA Passed

SDI-346

Verified – QA Passed

SDI-348

Verified – QA Passed

SDI-349

Verified – QA Passed

SDI-347

Verified – QA Passed

Outstanding Defects in Release 7.0.0


Invoice Extraction PWF

Defect ID

Title

Severity

37024

7.0.2/QAPaaS/Invoice Extraction - QnA model value was not showing candidate value for a some fields

3 - Medium

37425

7.0.2/QAPaaS/Invoice Extraction - Showing 'success' documents when user try to copy 'Failed' tab invoices in Run List

3 - Medium

37456

7.0.2/QAPaaS/Invoice Extraction - Run List success count is reducing when user deletes a file

3 - Medium

38117

7.0.2/QaPaaS/VAPT-Invoice Extraction - Broken Access Control

3 - Medium

38204

7.0.2/QAPaaS/Invoice Extraction: [SDI-503] Navistar - Inline item validation failure notification was too generic.

3 - Medium

38327

7.0.2/QAPaaS/Invoice Extraction - Advanced tab becoming Unresponsive when accessed(intermittently)

3 - Medium

36747

7.0.2/QAPaaS/Invoice Extraction - Runlist FAILURE status text showing is not unique

4 - Low

36871

7.0.2/QAPaaS/Invoice Extraction: Overlay panel for clear all line items is pointing to Company Address

4 - Low

38125

7.0.2/QaPaaS/VAPT-Invoice Extraction - Clickjacking

4 - Low

38126

7.0.2/QaPaaS/VAPT-Invoice Extraction - Exposed main.js File

4 - Low

38127

7.0.2/QaPaaS/VAPT-Invoice Extraction - Using Components with own vulnerability

4 - Low

Areas not tested

N/A

Known Issues / Limitations

Module

Limitations

Invoice Extraction PWF

  • Accuracy Issue associated with tesseract model, hence documents whose rotation angle is 180 will be inverted
  • Extraction is not possible for FOI which are in dark background
  • Accuracy issue for line item data
  • Invoice extraction in DU is failing intermittently with errors
  • When a batch is processed in DU there is a delay in reflecting the invoice count  in run list and updating the flags
  • 5 to 7 minutes delay is observed in updating accuracy analytics after the corrections
  • Advanced page hangs when project is having more than 700 invoices

Test Result Reference

Invoice Extraction PWF 

 # of User Stories

# of Test Cases

# of Open Defects (Critical/High/Med/Low)

Total

Fully
Passed

Partially
Passed

Total

Passed %

Critical

High

Medium

Low

Release 7.0.0

13

12

1

280

99.64%

0

0

1

0

Regression Release 7.0.0


140

95.00%

1

1

3

2

Invoice Extraction PWF

https://ustglobal.sharepoint.com/teams/InnovationEngineering/Shared%20Documents/Forms/AllItems.aspx?viewid=f349a736%2D8a62%2D467f%2D8448%2D067be464bd59&id=%2Fteams%2FInnovationEngineering%2FShared%20Documents%2FTechnology%2FDesign%20Review%20Council%2FE2E%20Docs%2F7%2E0%2E2%2FTest%20Cases%2FInvoice%20Extraction

ITOPsITOPs

Executive Summary

End to end functional and regression testing of ITOps is completed.

Period of Testing: 05th Oct 2020 to 04th Nov 2020

Test Execution Summary

Sl No.

Area

Execution Status

Open Defects if any

1

ITOps PWF

Completed

2 High defect open

Jira Defects Tested

Jira ID

Test Status

Comments

NA

VAPT Defects Tested

TFS ID

Test Status

Comments

NA

Outstanding Defects in Release 7.0

ITOps PWF

Defect ID

Title

Severity

38211

7.0.2/QA/ITOps : Filter- Within Days/Min/Hrs is not working

2 - High

38450

7.0.2/QA/ITOps : (SDI-515) Elapsed time is showing wrong values in the exported excel

2 - High

38512

7.0.2/QA/ITOps : Dashboard > Top 10 noise creating device - Resource name should be shown in Drill down widget, which user had selected.

3 - Medium

37806

7.0.2/QA/ITOps : MTBF for top Alerts is calculated for all alertnames instead it should be only for top 5 alertnames for that device. Also total Alert count to be shown in MTBF screen

3 - Medium

36962

7.0.2/QA/ITOPS: Project deletion throws error and deletion happens on second try

3 - Medium

37303

7.0.2/QA/ITOPS : Cluster list doesn't have all the same number of alerts in the cluster badge

3 - Medium

37475

7.0.2/QA/ITOPS: Severity colors are not reflecting immediately in the alert listing page

3 - Medium

38169

7.0.2/QA/ITOps : ihub lite channel configured to poll solarwinds data for every minute and sent to ihub main queue channel is not shown in ihub logs with polling for every minute.

3 - Medium

37718

7.0.2/QA/ITOPS: ITSM tickets with different priorities/severities created, come to our ES with corresponding priority number - but those are not mapped to severity colors

3 - Medium

37958

7.0.2/QA/ITOps : Getting duplicate project user's role in 'Add User 'section

3 – Medium

38255

7.0.2/QA/ITOPS: 'Assigned To' card in details page, even before they are assigned to a team member, has value.

3 - Medium

38244

7.0.2/QA/ITOPS: <Req bug> Alerts from same monitoring agent solarwinds but source name either as 'SW' or case sensitive values 'SolarWinds' - will not consider the severity mapping values and no severity added to ES

3 - Medium

38433

7.0.2/QA/ITOps : (SDI-538) Incorrect values for Created and Modified time in the exported excel sheet

3 - Medium

38122

7.0.2/QA/ITOps : VAPT - Low - Clickjacking - Application is vulnerable to UI Redressing attack.

3 - Medium

38123

7.0.2/QA/ITOps : VAPT - Info - Exposed main.js File - Application allow all the users to access the main.js file

4 - Low

38124

7.0.2/QA/ITOps : VAPT - Info - Using Components with known vulnerability

4 - Low

37799

7.0.2/QA/ITOPs : Dashboard - Open SNOW tickets- Specific message needed when no open tickets available

4 - Low

 

Areas not tested

QA validation is not done for Dashboard specific features like - creation and saving of new dashboard, edit, delete of dashboard, export option for widgets, add/manage options of widgets.

Known Issues / Limitations

Module

Limitations(Deferred Defects)

ITOps PWF

  • Dashboard - 'ticketEndTime' is not captured in the elastic search index and hence MTTR widget , Top 10 Alerts>Response time, Open Vs Closed(Top 10 Alerts) widget data is fully tested.
  • Dashboard - Time left displayed is not accurate in the SLA graph.
  • Dashboard Alert Pattern - With top 5 alerts for each day with different alert names coming in for different days, selecting large duration in date filter - X axis data goes not readable
  • Email Template - fields been missed in incoming alert template, data in other fields gets added with wrong data.
  • Email Verba Template - Service, Alert id, More information URL is not captured in Elastic search. Alert Attribute if not hostname OID, is not captured.
  • Email Template - Alert time not available in the template field, additional comma gets added to other Alert data in ES, resulting in failing to fetch inventory details
  • Email Template - Additional details if added to solarwinds mail format, gets added to previous field data in ES
  • Service now tickets created from ITOps, changing priority in SNOW is not changing severity of Alert back in ITOps.
  • Alert listing - The alert cluster pop-up pointer should be pointing to the correct cluster.
  • Alert listing - Applying severity on filtered result reset the advanced filter conditions.
  • Alert listing - Certain Alerts are not getting clustered when multiple alerts come one after the other. Alert will be in open status itself
  • Alert listing - More Field values are overlapped for longer text in the fields
  • Alert listing - Auto Refresh - No maximum limit validation available. Entering larger value does not work as expected.
  • Ticket Details Page is not getting refreshed with Assignment status
  • 'Assigned To' Card in Ticket Details page, does not show data when Assigned To field and Assigned Group field has values.
  • Configuration - Policies - Clicking on Done (primary button) is not saving the rule
  • Project name must be displayed in the alert listing page and infrastructure page
  • Extra whitespace in 9 dots

Test Result Reference

ITOps PWF 

 # of User Stories

# of Test Cases

# of Open Defects (Critical/High/Med/Low)

Total

Fully
Passed

Partially
Passed

Total

Passed %

Critical

High

Medium

Low

Release 7.0

96

96

0

866

98.8%

0

2

11

1

Regression Release 7.0

     

190

100%

0

0

0

0

 

ITOps PWF

https://ustglobal.sharepoint.com/teams/InnovationEngineering/Shared%20Documents/Forms/AllItems.aspx?csf=1&web=1&e=naljja&cid=504d3b2e%2Dafeb%2D42a7%2D92da%2Ddc4bbae661cf&FolderCTID=0x01200054818EE1119EB748AEF5796EDEBFECA1&viewid=f349a736%2D8a62%2D467f%2D8448%2D067be464bd59&id=%2Fteams%2FInnovationEngineering%2FShared%20Documents%2FTechnology%2FDesign%20Review%20Council%2FE2E%20Docs%2F7%2E0%2E2%2FTest%20Results%2FITOps

 

Core PlatformCore Platform

Executive Summary

End to end functional and TUC regression of Core Platform is completed.

Period of Testing: 5th Aug 2020 to 8th Oct 2020

Jira Defects tested

SmartOps platform

Jira ID

Test Status

Comments

SDI-109

Verified – QA Passed

SDI-96

Verified – QA Passed

SDI-95

Verified – QA Passed

SDI-269

Verified – QA Passed

SDI-338

Verified – QA Passed

SDI-417

Verified – QA Passed

SDI-338

Verified – QA Passed

VAPT Defects Tested

 

TFS ID

Test Status

Comments

31678

Closed

31680

Closed

 

Outstanding Defects in Release 7.0

Core Platform

Defect ID

Title

Severity

37553

7.0.2/QA/Core : The trace log view of skill is not seen completely

3 – Medium

36109

7.0.2/QA/Core Platform - iHUB- Queue channel disabled in ihub dashboard, still is working with messages pushed from rabbitMQ

3 – Medium

Outstanding defects CMS: Innovation Engineering - Test Summary Report - All Documents (sharepoint.com)

Areas not tested /Known Issues / Limitations
Not applicable
               

Test Result Reference

SmartOps Platform

 # of User Stories

# of Test Cases

# of Open Defects
(Critical/High/Med/Low)

Total

Fully
Passed

Partially
Passed

Total

Passed %

Critical

High

Medium

Low

 Release 7.0

52

52

0

474

98.16

0

0

2

0

 Release 7.0 Regression

     

286

98.95

0

0

0

0

Test Results                                  
https://ustglobal.sharepoint.com/:x:/r/teams/InnovationEngineering/_layouts/15/Doc.aspx?sourcedoc=%7B76E67D1A-C3F8-470F-B7A8-513D518C9D70%7D&file=TestResults_Rel%207.0.0%20CorePlatform.xlsx&action=default&mobileredirect=true

Feedback

Copyright © 2021 UST Global. All Rights Reserved.