ITOps v7.0.0- Post Deployment Process

An Installation Engineer is required to perform a few steps after SmartOps deployment is completed. The steps related to ITOps are:

Generate Offline Token

To generate an offline token for an organization, login to Master Organization and follow the steps below:

1. Click on Manage Tokens corresponding to the organization as shown in FigureFigure.

   

   This displays the token window as shown below

   

2. Click on the icon to copy the offline token.

Note: Ensure that you generate offline token for Ustglobal Organization.

Also ensure to inform the Deployment team about the new offline token, so that it can be updated in the key vault secret and containers will be restarted.

Hostnames (Updates) for configurations related to internal service invocation

In Kubernetes environment, the internal service names have changed when compared to Swarm environment. So, for internal service communications, all the hostnames of the services are now prefixed based on the domain. That is, based on the domain and region, the hostname will differ. This is applicable for areas where service URLS are configurable.

Example Scenarios:

• Clones Engine can now be accessed in SmartOps DI Platform internally as "http://smartops-dev-clones-engine:9090" (earlier this used to be http://clones-engine:9090). In QA, the prefix will change to smartops-qa. In PaaS ITOps DI, this will become http://itops-dev-clones-engine:9090 and so on.

• If you need to call any other service through micro action, you can follow this pattern.

• Another example is the case with elastic search Sense Channel where elastic search URL is configurable through UI. In such cases, Elastic Search URL becomes http://smartops-dev-elasticsearch:9200 for Platform DI environment.

Note: Only the name of the service has been changed as part of Kubernetes implementation, and no further changes in service side. Users can get the full list of applicable service names from Dev-Enablement team. 

Update Workflow shared Variables and reload cache

Workflow shared variables are certain variables which can be used in workflows. The default variables are saved in DB. Based on the installed environment, it is required to edit certain variables.

An installation engineer should ensure that the following variables are changed based on the environment. An API is currently available and can be used to edit the variables. After editing the variables, invoke another API to reload the cache. The details are given below:

API to edit variables:

Note:

• In Kubernetes based deployment, the variable value changes according to the domain name. So choose the correct domain names accordingly (the correct domain name for each installation can be checked with Dev Enablement team). The following table is an example for K8 SmartOps Dev environment for an Organization with ID, 1:

• As another example, In the ITOPs DI environment the variable value for clones.engine.host is: -clones.engine.host -http://itops-dev-clones-engine:9090

• To link XD project, ensure that you update the service ID of XD project, the details are:

• Similarly, if there are any other workflow shared variable that needs to be edited based on any change in client implementation, it can be done.

• If for an Organization ID, you are not sure about the Variable ID to be used, you can invoke the fetch API to fetch all the details of shared variables of an organization.

API to fetch all variables:

API to reload cache:

Clear Keycloak Cache

This step needs to be done only if clones-upgrade container was run for the deployment. To clear the Keycloak cache, follow the steps mentioned below:

1. Login to Keycloak Master Admin console.

2. Select any of the Realms and in Realm Settings and go to Cache tab.

3. Click on all the three Clear buttons (Realm Cache, User Cache and Keys Cache) to clear cache in all realms.

Note:

• This process is required only if the clones-upgrade program was run as part of deployment.

• In case the person who is doing the post-deployment steps are not aware of the deployment procedures completed, this can be executed.

PWF/Product/Component Association and De-association with Organization

For each environment, the de-association/association of PWFs, products, components in respective environments/instances (namely, core platform, ITOps, Invoice PWF) can be done. This will include the de-association of monitoring/support PWFs after data migration, as well as the association of new ITOps PWF and these steps are done via API.

Postman Collection contains sample request/response samples for the various APIs for association and de-association. It is on need basis that these need to get executed. For actual usage, these should be replaced with appropriate values for host, headers and request body. PFB the details of each of these APIs:

Postman collection available at: (https://ustglobal.sharepoint.com/:u:/r/teams/InnovationEngineering/Shared%20Documents/Knowledge%20Management/SmartOps%20Deployment/7.0.0/7.0_CorePlatform_PostmanCollection.zip?csf=1&web=1&e=UuTsda)

1. Invoke the SmartOps Login API for the applicable Organization.

   URL: POST https://<host_name>/pwf/api/smartops/login Body: {"realm":"<orgname>","userName":"<user_name>","password":"<password>"}

    

2. Invoke the Get all PWFS, Products and Components (stacks) API with the following parameters to fetch all available PWFS, products and components:

   URL: GET https://<host_name>/pwf/api/smartops/pwf/fetch-smartops-capabilities Headers: Organization-name–Unique identifier for the organization. Authorization –Access token generated after invoking SmartOps login API (offline token can also be used) user –User Id

   
   Note: This API response provides the details of the PWFs, products and components available and it is based on these values the input for the rest of the APIs given below can be derived.

3. Invoke the Add Product and PWF association API to add or associate applicable PWFs and Products in the required Organization.

   URL: PUT https://<host_name>/pwf/api/smartops/pwf/create-pwf-or-product-association Headers: Organization-name –Unique identifier for the organisation. Authorization –Access token generated after invoking SmartOps login API (offline token can also be used) user –User Id Body: {"componentId": <component_id>,"componentType":"<component_type>"} Component ID –Unique identifier of the package workflow or product (copied from previous step) Component Type –pwf / product

    

   
   Note: Repeat this step for every PWF / product to be associated to the organization.

4. Invoke Disable PWF Association API if you need to de-associate any PWF from an organization.

   URL: PUT > https://<host_name>/pwf/api/smartops/pwf/disable-pwf-association?pwfId=<pwf_id> Headers: offline-token or Authorization, user and Organization-key Param: <pwf_id>

   Note: Disabling PWF association will only disable the association of PWF with that organization. If there are any active projects associated wit that PWF, these projects should be de-activated or deleted to avoid further executions of these projects.

5. Invoke Remove Product Association API if you need to de-associate any Product from an organization.

   URL: DELETE > https://<host_name>/pwf/api/smartops/pwf/remove-product-association/<product_id> Headers: offline-token or Authorization, user and Organization-key Param: <product_id>

    

6. Associate/De-Associate components/stack to a Product or a PWF
   The details are as follows: 

   1. API details to Associate Stacks with PWF/Product:

      URL: PUT https://<hostname>/pwf/api/smartops/pwf/add-component-association Headers: offline-token or Authorization, user and Organization-key Body: {"componentId":<component_id>,"componentType":"<component_type>","stacks":[<list_of_stacks>]}

   2. API details to De-associate Stacks from PWF/Product:

      URL:DELETE https://<hostname>/pwf/api/smartops/pwf/remove-component-association Headers: offline-token or Authorization, user and Organization-key Body: {"componentId":<component_id>,"componentType":"<component_type>","stacks":[<list_of_stacks>]}

       

Note:

1. Replace request body with corresponding values of componentId, componentType, stack.

2. The componentType can be either product/pwf.

3. The componentId is the corresponding id of the product or pwf

4. The stacks should be list of applicable components which needs to be associated for the product/pwf.

5. The values of the componentId and the stack can be obtained from the API which fetches all PWFs, Products and Components (api/smartops/pwf/fetch-smartops-capabilities)

iHub – Configuration

<To be added>
‎

Archiva setup

Use below configurations to setup Archiva for proxy environments where network accessibility is not there:

For more on Archiva please refer:

http://clones-dev.southindia.cloudapp.azure.com:8080/smartops-dev-guide/archiva/

Creating Organizations and adding Users

• Organizations are the enterprise users for whom the SmartOps capabilities are offered.

• A Master Administrator has the privilege to manage organizations for SmartOps.

• Creating an Organization consists of the following sections

  • Create realm for a new organization

  • Defining Web Redirect URI for the Organization

Create realm for a new organization

To create a new organization in SmartOps, follow the steps mentioned below:

1. Access the Master User Interface. The Master UI login page is displayed as shown in FigureFigure.

   

2. Enter the user credentials (Master Admin) in the Username and Password field.

3. Click Sign in. SmartOps Master UI home page is displayed as shown in FigureFigure.

   

4. Click Organizations tab. By default, Organization is displayed.

5. Click on the icon near Organizations. The window for creating a new organization is displayed as shown in FigureFigure.

   

6. Enter the name of the organization in the Organization Name field.

7. Click Add. The SmartOps Master UI home page is refreshed with the new organization as shown in FigureFigure.

   

Defining Web Redirect URI for the Organisation

To define the web redirect URI for the new organization, follow the steps mentioned below:

1. Click on Edit Organisation corresponding to the organisation in FigureFigure. 

   
   This will open the Keycloak Administration UI where you can create an Organization Administrator user and do other configurations.

2. Click on Clients tab from the menu options available on the left-hand side of the screen and select smartops-frontend from the Clients as shown in FigureFigure.

   

3. Enter the URI in Valid Redirect URIs and Web Origins field as shown in FigureFigure.

   

4. Click Save. The success message is displayed as shown in FigureFigure.

   

Add Users and Assign Roles

Master Administrator can add users such as Admin, Installation Engineer, and Skills Designer through Master UI. The Roles and Permissions corresponding to each Package Workflow is detailed in the section Appendix: PWF Roles and Permissions To add users and assign corresponding roles, follow the steps mentioned below:

1. Click on Edit Organisation corresponding to the organisation as shown in FigureFigure.

   
   This will open the Keycloak Administration UI where you can create an Organization Administrator user and do other configurations.

2. Click on Users tab from the menu options available on the left-hand side of the screen and click Add User as shown in FigureFigure.

   

   The Add User screen is displayed as shown below.

   

3. Enter the username in the User Name field. You may also enter all other optional fields. Click Save.

4. Select the user from the list. You may search from the Search field.

5. Navigate to Role Mappings tab and add the required roles from the Available Roles block.

6. To add a role, select the required role from the Available Roles block and click on Add selected icon as shown in FigureFigure.

   
   

   You may assign roles as per requirement of the Package Workflow required as shown below. For example: Admin, Installation Engineer and Skills Designer Roles.

   

7. You may also assign password for the user. To assign password:

• Access Manage Password window through Credentials tab as shown in FigureFigure.

  

• Enter the password in Password field.

• Confirm the password in Password Confirmation field.

• Mark the password as temporary or not in the Temporary field.

• Click Reset Password. The password reset successfully message is displayed.

ITOps roles

• itops_admin

• itops_engineer

• customer_visitor

Removing Unwanted Roles and Permissions

To remove unwanted roles and permissions, refer Removing Unwanted Roles and Permissions

Import Micro-actions to Organisation Environment

Since 7.0 is a fresh install, ITOps workflows and microactions need not be imported. The complete list of workflows and microactions are given in the appendix for reference:

Keycloak Mapper Configuration

Perform Keycloak Mapper configuration to enable role based access.

1. Login to Keycloak admin console

2. Select the realm and click on ‘Clients’ from the left navigation pane

3. Add a new protocol mapper to the client "Smartops-frontend"

4. Set the Mapper Type to User Realm Role

5. Set Token Claim Name as role

6. Set Claim JSON Type as String

7. Enable Add to userinfo & Save

Reference Data updates to be made and API details

Review the data in ITOpsReferenceData Mongo collection. APIs mentioned below can also be used to view all the data.

https://smartops-k8s-dev.eastus.cloudapp.azure.com/paas/itops/alertmapping/swagger#/Reference%20Data/fetchAllReferenceData

Any changes to the data can be made using the add and delete APIs provided below:

https://smartops-k8s-dev.eastus.cloudapp.azure.com/paas/itops/alertmapping/swagger#/Reference%20Data/Deletereferencedata

https://smartops-k8s-dev.eastus.cloudapp.azure.com/paas/itops/alertmapping/swagger#/Reference%20Data/insertReferenceData

Note: assignToGroups, holdState, closeState, timezone etc are some of the values that could be changed based on installation environment.

Creating/Managing ITOPS Project

1. Create ITOps project from UI. The ITOps project configuration consists of the 2 sections below.

   • General Configuration

     • Response SLA Threshold – The maximum number of tickets in unassigned state after which it is considered as breach.

     • Service Now Host and Credentials

   • Scheduler Configurations

     • Scheduler for flap clusters – Closes flap clusters in which no new alerts are getting added after the configured time interval.

     • Correlation Scheduler – Clustering of alerts and ticket creation

     • Ticket Status Update Scheduler – Keeps the ticket and alert details in sync with ITSM

2. Configure Correlation and Acknowledgement policies and rules from ITOps Configuration screen.

API driven configurations for the ITOps project

Setting Recovery Policies

Policies can be added/edited/deleted using below APIs

https://smartops-k8s-dev.eastus.cloudapp.azure.com/paas/itops/alertmapping/swagger#/Policies/addAlertPolicy

https://smartops-k8s-dev.eastus.cloudapp.azure.com/paas/itops/alertmapping/swagger#/Policies/updateAlertPolicy

https://smartops-k8s-dev.eastus.cloudapp.azure.com/paas/itops/alertmapping/swagger#/Policies/deleteAlertPolicy

Policies can be retrieved using below API.

https://smartops-k8s-dev.eastus.cloudapp.azure.com/paas/itops/alertmapping/swagger#/Policies/fetchAlertPolicieswithtype

Note: Policy Type = REC for recovery.

Setting Recovery Rules

Rules can be added/edited/deleted using APIs @ https://smartops-k8s-dev.eastus.cloudapp.azure.com/paas/itops/alertmapping/swagger#/Rules

Setting  Auto closure and flap conditions

Below APIs can be used to add/delete/fetch autoclosure and flap conditions:
https://smartops-k8s-dev.eastus.cloudapp.azure.com/paas/itops/alertmapping/swagger#/ Autoclosure%20and%20Flap%20APIs/saveAutoClosureConditions

https://smartops-k8s-dev.eastus.cloudapp.azure.com/paas/itops/alertmapping/swagger#/ Autoclosure%20and%20Flap%20APIs/deleteAutoClosureConditions

https://smartops-k8s-dev.eastus.cloudapp.azure.com/paas/itops/alertmapping/swagger#/ Autoclosure%20and%20Flap%20APIs/fetchAutoClosureConditions

Inventory Import

Use below API to import inventory from an excel file with inventory details
https://smartops-k8s-dev.eastus.cloudapp.azure.com/paas/itops/alertmapping/swagger#/Device%20Inventory/Importdeviceinventory

Sample file is available at: https://ustglobal.sharepoint.com/:x:/r/teams/InnovationEngineering/Shared%20Documents/Knowledge%20Management/SmartOps%20Deployment/7.0.0/Asset_inventory.xlsx?d=w758237471c8c4a44bf68264b014e2c3e&csf=1&web=1&e=Kt5FZ1

Topology Import

Use below API to import topology from an excel file with topology details
https://smartops-k8s-dev.eastus.cloudapp.azure.com/paas/itops/itopscorrelation/swagger#/Import%20Export/importtopology-Excel
Sample file is available at:

https://ustglobal.sharepoint.com/:x:/r/teams/InnovationEngineering/Shared%20Documents/Knowledge%20Management/SmartOps%20Deployment/7.0.0/Sample_Topology.xlsx?d=w74ae4b3857bc409aad513a8fdaa4ff7b&csf=1&web=1&e=puzStE

HUB Channel Configurations

API/Queue Channel for Alerts

1. Select the checkboxes to include Request received time and to send alerts as list.

2. Add the below fields in default section  

   1. automationStoryName = PwfITOpsRealtime

   2. senseParams/timezone = the timezone of the time fields in the alert message

   3. senseParams/dateFormat = the date format of the time fields in alert message.
      Note: Fields accepted by ITOps in alert message given in appendix. The incoming alerts should be transformed into these fields. New fields can be added by using the index update API in the index.

Email Channel for Alert

1. Select the checkboxes to include Request received time and to send alerts as list

2. Enter automationStoryName as PwfITOpsExtractEmailAlerts
   

API Channel for Snow Tickets

1. Select the check-boxes to include Request received time and to send alerts as list.

2. Add the fields mentioned below in default section

   1. */senseParams/source = itsm

   2. */automationStoryName = PwfITOpsAddAlerts

   3.  JSON spec needs to be as per below screen shot:

Recommendation from dev team - In SNOW channel, fetch tickets created after the last poll to reduce number of tickets being polled and reduce the load on the system. 

Ticket Status Update Scheduler

This is the scheduler which keeps the ticket/alert details in the ITSM tool in sync with the ITOps Alert Store. By default, the workflow gets tickets updated in last 30 mins and created by the user given in project configuration. Any changes to the filter conditions should be made by editing the settings of the workflow and/or changing the microactions used. Details are as below:

• Changing the default 30 mins interval – Set the required value(in mins) in the field ‘time_interval’ in workflow settings for the workflow ITOps_PWF_TicketUpdateScheduler

• Adding different filter conditions in the query from INCIDENT table – Edit ‘url’ field in 'RetriveInciidentDtl' microaction in ITOps_PWF_TicketUpdateScheduler

• Adding different filter conditions in the query from TASK SLA table – Edit ‘url’ field in 'RetriveSlaDtl' microaction in ITOps_PWF_TicketUpdateScheduler

Customisations in E-mail Microactions

There are 2 sources available now solarwinds and verba. The time zone and dateformat for them are as below :
‎

If these need to be changed, the following fields should be changed in the workflow settings of the workflow ITOps_PWF_ExtractEmailAlerts

• solarwindsTimezone

• verbaTimezone

• solarwindsTimeFormat

• verbaTimeFormat

• verbaEmail

• solarwindsEmail

ITOps-Grafana Deployment Activities

ITOps Grafana deployment activities must consists of the following three activities

• Keycloak integration

• User and Role integration 

• Integration of dash board using the JSON import

KeyCloak Setup for Grafana

• Login in to Keycloak. URL: <domain>/paas/itops/keycloak/auth/admin/master/console/

• Select the Realm

• Select Clients from the left side Navigation

  

• Choose ‘Create’ for creating a new Client.

• Provide details as below and choose ‘Save’. Make sure to opt client-protocol as ‘openid-connect’.

• Select the created client (grafana_client) from the client list screen

• Change the Access-type value as ‘confidential’ from the settings screen.

• Add * to the valid redirect URIs and Web origins list and save the settings

Example: https://smartops-k8s-dev.eastus.cloudapp.azure.com/paas/itops/*

• Choose the ‘Credentials’ tab

• Select the ‘Client Authenticator’ as Client Id and Secret

• Below a secret client token will be auto generated.

• This secret client token and client id with Grafana Config setup to connect with keycloak as GAA.

Grafana UI  Configuration with KeyCloak 

• The configuration can be updated using an Environment var with Kubernetes / swarm.

• Add the client id and client secret created from the key cloak in to the Azure Key Vault (For adding to contact DE team).

  • Client_id – grafana_client

  • Client_secret – 233444411****

• Restart the server and the Grafana authentication will be pointing to the respective Key cloak page for login.

User and Role Configuration

Admin User:

To enable the admin login in Grafana provide the email id as admin@localhost in key-cloak for the admin user.

Other Users:

To enable the user login for the user other than admin (admin@localhost), add the user details in the Grafana user list from dashboard itself. There are three types of organization roles in Grafana:

• Admin - For managing data sources, teams, and users within an organization.

• Editor - For creating and editing dashboards.

• Viewer - For viewing dashboards.

For adding the user details please following step.

• On the sidebar, click the Server Admin (shield) icon.

• In the Users tab, click New user.

• Enter the following user details

  • In Name, enter the name of the user.(eg: itops_adimin)

  • In E-mail, enter the email of the user (eg: itopsadmin@localhost). Keycloak and Grafana email should be same In Username, enter the username.

  • In Password, enter a password.

• Click Create to create the user account.

Integration of Dashboard using the JSON Import 

Update ‘apiURL’ in the Dashboard JSON:

In ‘ITOps_Dashboard.json’ file change the ‘apiURL’ value of all (10) graph widget with the production elastic wrapper api URL.

Download the below zip file for ‘ITOps_Dashboard.json’

from the location

Example:

• Import the ‘ITOps_Dashboard.json’ in to Grafana After the changing the ‘apiURL’ value import the JSON in to the Grafana. For importing the JSON follow the below step.

• Click on the Mange Dashboard menu from the left side bar as shown in the below screen shot.

• Click on the import button from the manage dashboard page

• When click on the import button a new page will be shown with the button ‘Upload JSON file’. Click the ‘Upload JSON file’ button for selecting the ITOps_Dashboard.json file then click on the ‘import’ button for saving the JSON.

• Enable the Data Source Plugin- For connecting with API:
  Use the ‘wrapper-api’ data source plugin for connecting the widgets with the API. For enabling the data source plugin, follow the below step:

• Click on the Data Source menu from the left side bar for going to the Data Source configuration page.
  

• From the configuration page click on the ‘Add data Source’ button for listing all the available data source plugin. For enabling the plugin, In the search box enter the text 'wrapper-api', from the search result select the wrapper-api plugin and enable it by clicking the select button for connecting API.

Appendix

Fields in alert message accepted by ITOps:

• alertName,

• alertMessage,

• alertTime,

• severity,

• nodeName,

• ipAddress,

• objectName,

• objectStatus,

• objectType,

• resourceType,

• alertMetric,

• alertDetailsURL,

• source,

• portNumber

API to add fields to alert index

https://smartops-k8s-dev.eastus.cloudapp.azure.com/paas/itops/alertmapping/swagger#/Index/addFieldToIndex

Workflows and Micro actions developed for ITOps

Workflows

Microactions