AIOPS v.2.6 - Application Deployment Process‎

Overview

SmartOps product deployment configuration has been revised to support better flexibility and configuration. Products will be packaged in new format referred as “Smart package” and will be deployable using tool “smartpkg”. This document covers instructions to migrate existing environment configuration to new format and installation of AIOPS 2.5.2 using new mechanism. Same set of instructions can be followed hereafter except the migration steps, which applies only for first time.

Prerequisite

Download the tar file from aiops-2.5.2.tar.gz

Preparation and Post-Deployment Activity

To meet the solution stack prerequisites and requirements.

1. Validate and Upgrade smartpkg package version.

2. Kubeseal Installation in Bastion Server

3. Sealed Secret Controller Installation

4. Backup and generate sealed secret from the existing secret.

5. Take a backup for all the helm manifests

6. Delete helm manifests
   ‎

Validate and Upgrade smartpkg package version.

1. Login to the VM Server

2. Validate if the smartpkg has installed and if they use the latest version. Run command below.
   ‎$ python3 -m pip list | grep smartpkg
   ‎
   ‎Note: Proceed step #3 if smartpkg version is not latest version (0.0.17). If the smartpkg has not yet installed in the VM please proceed the installation of Smartpkg Installation in bastion‎

3. Uninstall the smartpkg library. Run command below.
   ‎
   ‎$ python3 -m pip uninstall smartpkg==0.0.14
   ‎

4. Install the smartpkg library with the latest version.
   ‎
   ‎$ python3 -m pip install smartpkg==0.0.17 -i https://3thxl5c3tscnjmg3cxjwmc2lnw2i7gwtug7xookemzo6mmrutzcq@pkgs.dev.azure.com/USTInnovationEngineering/SmartOps/_packaging/python-lib/pypi/simple  --  
   ‎

Smartpkg Installation in bastion

Smartpkg is a python-based tool to help managing and deploying smartops Product packages based on helm charts. This is a prerequisite to deploy smartOps products. SmartPkg should be installed in machine from where the Kubernetes deployment happens which is usually a bastion machine.

Below are the pre-requisites for installing smartpkg 

1. Python3 should be installed in the machine  

Installation Steps

1. Login to the machine you want to install smartpkg 

2. Execute => sudo apt-get install python3-pip

3. Execute => python3 -m pip install ConfigArgParse wheel pyyaml jsonschema

4. Execute => python3 -m pip install smartpkg==0.0.17 -i https://3thxl5c3tscnjmg3cxjwmc2lnw2i7gwtug7xookemzo6mmrutzcq@pkgs.dev.azure.com/USTInnovationEngineering/SmartOps/_packaging/python-lib/pypi/simple  --  

5. Verify the installation by => smartpkg –help
   ‎

Note: If smartpkg: command not found error shows up, logout and login or open a new session. 

kubeseal Installation in Bastion Machine

1. Download the binary file from github.‎
   ‎

2. Change the file mode permission and verify the version by running the command.‎
   

 

Sealed Secret Controller Installation

1. Add the smartops-helm-virtual Helm repository to your Helm client.
   
   ‎
    
2. ‎‎Update the repository.
   
   ‎
   ‎
3. Create namespace.
   ‎
   ‎
4. Install the chart with the release name sealed-secrets-controller.
   
   ‎
   ‎‎
   ‎
5. Validate and find the name of the pods using this command.
   ‎
   ‎
   ‎
   ‎

Backup and generate sealed secret from the existing secret

1. Create a folder and switch to that folder.
   
2. Get a copy of the existing secret to a file
3. Seal the secrets using kubeseal command:
   
   ‎Note: Please make sure to include extra secrets added for each Release in env_secret.yaml file before it is sealed using kubeseal command.

   ‎

4. Create a file “env_secrets.yaml” and Copy/Paste the generated sealed-secrets to the external configuration file in env_secrets.yaml.
   ‎
   ‎Here is how it looks an example.
   ‎
   ‎

‎Take a backup for all the helm manifests

1. Go to directory folder and create backup folder.
   
   ‎‎

2. Fetch the manifest of all the helm releases using below script (Please substitute namespace and kube-context wherever required)
   
   ‎‎‎

3. Provide execution permission and execute manifest.sh
   ‎
   ‎

4. Fetch manifest for logmon which is in different namespace
   ‎
   ‎

Delete helm releases

Note: Please make you’ve backup the sealed secrets.
‎

1. Execute the script to uninstall all the releases from releases.txt file

   delete_release.sh

   #!/bin/bash

   while read line; do helm uninstall $line -n itopsv1-stg01 --kube-context= kc-smartops-itops-stg; done < releases.txt‎

2. Provide execution permission and execute delete_release.sh
   ‎$ chmod +x delete_release.sh

   $ bash delete_release.sh‎

3. Uninstall all the remaining releases‎
   ‎$ helm uninstall smartops-logmon-logmon-rel -n smartops-logmon --kube-context kc-smartops-itops-stg
   $ helm uninstall sealed-secrets-controller-rel -n sealed-secrets-controller --kube-context kc-smartops-itops-stg
   ‎$ helm uninstall smartops-common-secret-setup-rel -n smartops-common --kube-context kc-smartops-itops-stg

 

 

Application Deployment

Create product and env files
‎

1. Go to the directory folder.
   
   ‎‎
2. Upload the compress file in the Bastion server. Please find the file at email attachment.
   
   ‎‎
3. Copy the “env_secrets.yaml” file and place the file to values folder.
   
   ‎‎‎
   ‎Example folder structure.
   ‎
   ‎
4. Run the smartpkg deploy command from the path where values folder and product.yaml file are present.
   
   ‎‎‎
5. Validate if all pods are running.
   ‎
   ‎