Platform v6.4.3- Deployment Process (Docker Swarm)

Purpose of this document is to provide guideline for deploying SmartOps Platform in a docker swarm environment for azure. The document contains prerequisites for installation and detailed instructions to deploy SmartOps along with FAQs and post deployment validation steps.

Setup Docker Swarm Cluster

Follow the below instructions to set up docker swarm environment and to complete the prerequisites for deploying SmartOps application.

Deployment Timings

Timings will vary based on the environment and network

 

Preparation Activities

Following are the preparation activities:

Setup Resource Group and Networks

• Before beginning the VM setup it is always better to plan the Resource group and network in Azure so that we have better control over security and usage.

• The above architecture diagram provides guidance on setting up a Virtual Network and a Subnet for the installation. If deviating from above plan, consider applying the effects of that change in subsequent steps of this document.

• Also, for better cost control and auditing, place all resources in an installation under one resource group.

Creating Docker Swarm Node VMs

• Create Node VMs with below configurations to create the infrastructure from Deployment VM

• Number of VMs required for installation is subjective to the stacks that are required for the installation at hand. More details on sizing will amended as required in subsequent releases.

• Recommended hardware details are mentioned below.

 

 

*Accelerated Networking should be enabled in VM NICs for better performance of cluster.

Create the VMs with authentication mechanism as SSH public key

 

https://docs.microsoft.com/en-us/azure/virtual-machines/linux/quick-create-cli?toc=%2fazure%2fvirtual-machines%2flinux%2ftoc.json

 

• Considering the Azure resource group may differ in environments, it is necessary to log on to azure cli and set the default resource group to the one used for this purpose (Steps to Install azure cli is available in FAQ)

• Place the public key file to create VM from where the script is getting executed. The key is passed as parameter ssh-key-values and the key file name is smartops-dev-env-public-compact in script Eg: --ssh-key-values @smartops-dev-env-public-compact

• The following values should be updated correctly in the Script. Please not the script is used to create VMs with naming conventions used by SmartOps Engineering Team

 

Place install_dependencies.sh and docker-setup.sh scripts and public key file in same location where the vm creation script(smartops-suite-vm-setup.sh) is being executed. install_dependencies.sh and docker-setup.sh files will install all prerequisite softwares required to deploy the application. The failure in execution of this script causes failure in installation of SmartOps.

 

VM Names are set as vm-ENV_PREFIX-ENV_ID-PURPOSE-<Sequence>

 

Refer the above script with adequate input parameters to create 3 VMs with the names as below. The names are just for reference. Team can decide the naming convention of VMs to be followed.

vm-smartopsengg-dev-smartops-app-001, vm-smartopsengg-dev-smartops-app-002, vm-smartopsengg-dev-smartops-infra-001

As the model files are installed from jumpbox for previous deployments, please make sure that azcopy is installed in App1 VM before installation.

Instructions for Proxy Enabled and Client Environments

• Proxy in docker daemon should be added and set up openSSL certificate in Proxy enabled and client environment

• Follow below steps for adding proxy in docker daemon. Edit proxy server and port number accordingly for the environment

mkdir -p /etc/systemd/system/docker.service.d sudo vi /etc/systemd/system/docker.service.d/http-proxy.conf #Add below configuration if not there already by replacing the proxy server and port for the environment. [Service] Environment="HTTP_PROXY=http://172.27.71.42:9401" Environment="HTTPS_PROXY=http://172.27.71.42:9401" sudo systemctl daemon-reload sudo systemctl show --property Environment docker sudo systemctl restart docker

 

• Ensure that Open SSL Certificate (by default it will be /usr/local/share/ca-certificates) is updated.

• Ensure that Access to below Docker Registry is in place

  • smartopsregistry.azurecr.io

  • smartopdevsregistry.azurecr.io

• Request help from respective Cloud Admin Teams to apply the certificates and add registry access in the environment

  Initialize Swarm Cluster

• Docker Swarm will be initialized as part of install_dependencies.sh script by default.

# To initiate swarm cluster, run the following command: docker swarm init

 

• Appropriate connectivity (SSH) needs to be established between Deployment VM and Cluster VMs (docker swarm nodes).

• Connect to the APP-001 VM through ssh authentication and get swarm join token in App VM

ssh <APP-001> docker swarm join-token manager # To add a manager node to this swarm, run the following command: docker swarm join --token SWMTKN-1-2g2y0i29v726ephzrnrvcw0wnbhj0wnnos90adxepoajlfda08-btqj81s4j30xupd8gaf2tzesm 172.27.0.14:2377

 

• SSH in to other VMs and execute the copied command from previous execution to join as a manager

Note: Find more info about setting up a docker swarm cluster here and other links in the website.

ssh <App-002 VM> docker swarm join --token SWMTKN-1-2g2y0i29v726ephzrnrvcw0wnbhj0wnnos90adxepoajlfda08-btqj81s4j30xupd8gaf2tzesm 172.27.0.14:2377 exit ssh <Infra-001 VM> docker swarm join --token SWMTKN-1-2g2y0i29v726ephzrnrvcw0wnbhj0wnnos90adxepoajlfda08-btqj81s4j30xupd8gaf2tzesm 172.27.0.14:2377 exit

• Execute below command from manager node to identify the docker nodes

docker node ls Result: ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION oout8h6cf8z42pu03xb44xv66 * vm-smartopsengg-dev-app-001 Ready Active Leader 19.03.8 l39k1xair19wfm225bphuslpg vm-smartopsengg-dev-app-002 Ready Active Reachable 19.03.8 z08tkuqp232tb5kjp4c87ce5g vm-smartopsengg-dev-infra-001 Ready Active Reachable 19.03.8

• Download deployment artifacts smartops-platform-R6.4.3.tar.gz from the SharePoint location and copy the deployment artifacts to one of the Docker Swarm Manager Node’s home directory. [ ~ ]. (After creating VM using above steps)

  Label Swarm Cluster Nodes – (Automated steps- For information of Installation engineers)

• Labelling is an important step in setting up the cluster. The node labels ensure that the docker stacks are scheduled in the correct VMs as per the Node planning that we have done above. Also the deployment architecture described in SmartOps in Docker Swarm design document is implemented using node labeling

• Nodes must be labelled for the following

• Infrastructure and Log Monitoring Node

• App and Database Node Label (For Database node): - This is required as we are giving node affinity for databases, so that each database gets scheduled on particular node.

• Labelling of Nodes are handled by the deployment script for SmartOps based on details provided in config.ini file.

Infra Node fix for Elasticsearch - (Automated steps- For information of Installation engineers)

Apply below command to Infra Node VM if not reflected already. This is required for Elasticsearch

log in to the Infra VM (where the elastic search has installed) and validate if max map count is set

Reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html

sudo sysctl -w vm.max_map_count=262144 #view the content of /etc/sysctl.conf and execute only if below entry is not present. ‎#viewing the content of /etc/sysctl.conf ‎ ‎ cat /etc/sysctl.conf sudo bash -c 'echo "vm.max_map_count=262144" >> /etc/sysctl.conf'

 

Configure Azure Application Gateway – SmartOps

• Create an application gateway in azure portal following instruction from Azure

Reference:

https://docs.microsoft.com/en-us/azure/application-gateway/overview

https://docs.microsoft.com/en-us/azure/application-gateway/quick-create-portal

• This document will provide details on how to set up and configure application gateway for SmartOps

 

Pre-Deployment Activity

Deploy SmartOps

• smartops-platform-R6.4.3.tar.gz contains the artifacts for SmartOps Platform. Extract the deployment artifacts using following command:

tar -xvf smartops-platform-R6.4.3.tar.gz

• Navigate to installer directory in the package.

cd <package_name>/installer/

• Execute initialization script

sh init.sh

 

• Edit config.ini file and update environment specific values as per table

vi config.ini

 

 

 

• If it is proxy enabled environment, execute below command to replace the proxy variables

Eg: sed -i "s,gateway.zscaler.net,http://172.67.17.50,g" config/*/*.env

sed -i "s,PROXY_PORT=9400,PROXY_PORT=8500,g" config/*/*.env

 

cd ~/<package_name>/stack-repo/ sed -i "s,PROXY_ENABLED=false,PROXY_ENABLED=true,g" config/*/*.env sed -i "s,gateway.zscaler.net,<PROXY_SERVER>,g" config/*/*.env sed -i "s,PROXY_PORT=9400,PROXY_PORT=<PROXY_PORT >,g" config/*/*.env

 

Execute the following command to start deployment. The deployment takes 60+ minutes to complete

cd ~/<package_name>/installer/ ./deploy.sh

 

Note: When deploying the services for first time, all the images are getting downloaded from image registry. So, there could be failures/delay in startup. If you see certain services failing after waiting for a period, you can try to redeploy using stop-start-stacks.sh for those specific stacks present in <package_name>/ stack-repo, or force update specific services. E.g is shown below.

./stop-start-stacks.sh <stack-name> docker service update <stack-name>_<service-name> --force

 

Refer Post Deployment Checks & Validations for more validation steps on deployment

Deploy Log Monitoring Stack

• Navigate to ‘<package_name>/stack-repo/env-setup’

• Edit config.ini file with the values as below.

 

• execute the script install-env.sh to set up the monitoring stack

./install-env.sh

• Verify the deployment using following commands

docker stack ps smartops-logmon docker service ls | grep smartops-logmon *** To know the service is up and running, use below command docker service ps <service_name>. service name is available from above command ***** docker service ps smartops-logmon_alertmanager-monitor

 

• Wait for 10 minutes after deployment for Kibana to create indexes

• To access the logs from kibana use the following URL

 

Post-Deployment Activity

Additional Config Steps in Deployment

It is necessary to add the domain and Active Directory (if enabled) in proxy to allow the access via Application

Once app-gateway is up and running you can access the application in browser with the url >/

Before starting to use the application, certain config steps are to be performed, which includes application restarts. Following config steps are to be done.

1. Check keycloak configuration pages and add a new user.

2. Generate New Offline token

3. Update the stacks with new token and redeploy app stacks

Check and Update Keycloak Config, add User and Protocol Mapper

1. Access keycloak URL with following link in a browserExample: https://smartops-prod.eastus.cloudapp.azure.com/keycloak/auth/

2. Login to Administration Console using userid/password as admin/admin

3. Go to “Clients” menu item on left pane and select smartops-frontend

4. In Validate Redirect Uri you should see (if not enter it manually).

5. In Web Origins add the following uri

6. Click Save

7. Go to “Users” menu item on left pane.

8. Click Add User button

9. Enter username as “smartops” and click save.

10. Go to “Credentials” tab for the user and enter a password for login

11. Go to “Role Mappings” tab for the user.

12. For the time being assign all the roles to the user by selecting all from left list and putting it to right.

Create Keycloak Protocol Mapper

To create a protocol mapper named “role”, select the realm in dropdown, Clients in Configure, click on Smartops-frontend and click on Create in Mappers tab.

Provide the values mentioned in below screenshot and click on Save

 

Get Offline token and update the stack files

• Go to in a browser.

• You will get the login page.

• Enter admin/admin as credentials

• Once logged in select multi menu from right top of the screen and click on “Creator’s Studio”

• You should see the Organizations tab on screen now.

• Click on “Manage Tokens” in UST Global Organization. Copy the existing token. Keep the token in a file so that it can be referred later.

Update the stacks with new token or NER Service ID and redeploy app stacks

• SSH into a manager node.

• Navigate to ‘<package_name>/installer’ directory.

• Update OFFLINE_TOKEN in config.ini file and save it. (You can also update NER Service ID in same way)

• Execute the script to replace the stack files with updated token.

• Navigate to ‘<package_name>/stack-repo’ directory.

• Re-deploy Smart Sense, DU and Monitoring Stacks.

cd <package_name>/installer/ vi config.ini # Update value for OFFLINE_TOKEN variable and save the file. ./update_additional_values.sh cd ../stack-repo/ ./stop-start-stacks.sh smartsense smartops-du monitoring

 

Setting up Indices for Kibana Logs

After opening Kibana, choose the Logs card on the home page. Move to the Settings tab and go to the Log Indices section to set the value as shown below.

To review the container logs via kibana, we have to set up the log indices pattern to “logstash-*”.

Configure Azure Load Balancer for External access of RabbitMQ in TLS enabled channel

• Create an Azure Load Balancer for external access of RabbitMQ .Please follow the attached document and configure an Azure Load balancer

• Configure appropriate Rule in Load Balancer to point it to back end pool VM.(Add Infra VM -001 as the back-end pool for Smartops application)

 

Post Deployment Checks & Validations

Validate Deployment

• On successful installation , services in attached list will be deployed in cluster. The installed service can be found out using following command

docker service ls

 

• Once Deployment is completed verify the services to check whether all required services are up and running. Please not that below services with name suffix as ‘patch_update’ and ‘liquibase’ will be down as they are one-time containers.

docker service ls | grep 0/1

 

• When manager node is running with high Network Usage and to deploy a stack with high performance services at same time causes the following issue of ‘Error respose from Deamon: RPC error’. Deployment Scripts deploys stack one after another with 30 second interval, there is a possibility of request gets queued and throttle network during that time. Execute “docker service ls | grep 0/1” which will list out the containers which are either stopped or being prepared to start.

 

 

Restart respective stack using stop-start-stack.sh script which had rpc error.

Example: contract-mgmnt_cms_doclink_process is down. Then execute following steps to restart it .

The stack name will be ‘contract-mgmnt’ for above service.

cd ../stack-repo ./stop-start-stacks.sh contract-mgmnt

 

 

• Validate the status of stack by executing “docker stack ps <stack_name>” which will then provide the list of services of that stack with its current status

docker stack ps <stack_name>

 

Perform Post Deployment Configurations

Before starting to use the application, certain config steps are to be performed. Refer Post Deployment-Configuration Guide for executing the post deployment activity steps for configuring PWF roles and permission. If it is already executed, please ignore the steps.

Note: This step is mandatory to view the components available in the Home Page of application and in menu (nine dots)

Validate Application Access

Validate Application access by accessing following URL and login using the user id/password

https://<app-dns-name> /

 

 

URLs for accessing application features

Replace <app-dns-name> with the dns name configured in application gateway.

Eg: smartops.eastus.cloudapp.azure.com

For broker oidc endpoint:

https://<app-dns-name>/keycloak/auth/realms/ustglobal/broker/oidc/endpoint

For accessing SmartOps Application:

https://<app-dns-name> /

For accessing Master Realm for monitoring Application:

https://<app-dns-name>/uimux/smartops/master

For Logout URL:

https://<app-dns-name>/keycloak/auth/realms/ustglobal/protocol/openid-connect/logout

For Accessing logs:

 

Archive Release artifacts (Production or Client Environments)

The deployment artifacts contain sensitive information such as passwords . Hence the deployment folder should be archived to a protected location and removed from the VM after Successful Deployment.

# acrhive the deployment folder cd ~ sudo tar -pcvzf <package_name>_archive.tar <package_name> # Move created archive to a secured location # Remove deployment folder sudo rm -R <package_name>

AD Integration

Please follow attached document below to integrate keycloak with an Enterprise AD.

 

 

Feedback

Copyright © 2021 UST Global. All Rights Reserved.