ITOps v1.3.0-Infrastructure Deployment Process

Important Terms
‎

Overview

Welcome to Kubernetes Infrastructure deployment guide for PWF ITOps 7.2. This document provides an overview of the flow of activities and overall architecture for installation of PWF IE 7.2
The main activities involved in this Kubernetes Infrastructure deployment guide is as below. 

• Base Infrastructure Installation in Azure Cloud

• Installation and Configuration of Kubernetes Infrastructure for PWF ITOps 

To get a deeper understanding about the installation read further about SmartOps 7.1 architecture.

Architecture ‎

PWF ITOps 7.1 is a distributed cloud application. It can be installed only in cloud infrastructure.  PWF ITOps 7.2 is deployed and managed in Kubernetes. Even though it is there in our roadmap to be cloud agnostic, at this point of time the installation is compatible and tested only on Microsoft Azure Cloud Infrastructure. Support for more cloud will follow in subsequent releases. 
The below diagram provides an overview 

• The installation is done on a single virtual network with multiple Subnets.  

• Each Subnets will be self-isolated with NSG rules and only required access granted. Each PWF and Platform will reside in its own Subnet.  

• Additionally, there will be a maintenance subnet where a bastion is hosted for administrative activities. 

• In certain situations, clients of SmartOps PWFs will have to get TCP connectivity to assets in SmartOps. For e.g. Posting data to a SmartOps Queue. In such cases as the diagram shows a VPN connectivity is established from client network to SmartOps Network. The connectivity is terminated at a DMZ subnet where a squid machine will isolate client network from SmartOps application subnets, for enhanced security. 

Prerequisites set up before downloading the package and deploying the infrastructure

Download the installation package in cloud shell

The Release packages are stored in SharePoint location and in azure artifacts. Please follow below steps for downloading the Azure Infrastructure installation scripts to Azure cloud shell.

Open Azure cloud shell

‎

az login in cloud shell

The package downloaded from SharePoint has to be made available in Azure cloud shell
‎

SharePoint location

https://ustglobal.sharepoint.com/teams/InnovationEngineering/Shared%20Documents/Forms/AllItems.aspx?viewid=f349a736%2D8a62%2D467f%2D8448%2D067be464bd59&id=%2Fteams%2FInnovationEngineering%2FShared%20Documents%2FKnowledge%20Management%2FSmartOps%20Deployment 

After downloading the package in cloud shell scripts for deploying

Scripts location for Base Infra and Product Infra deployment

After package is downloaded, the scripts for installing base and product infra will be available in

<package_folder_name>/azure-setup/scripts folder

ARM Templates for Base Infra and Product Infra deployment

After package is downloaded, the scripts for installing base and product infra will be available in

<package_folder_name>/azure-setup/arm-templates folder.

Install Base Infrastructure

• Once the package is downloaded, cd into <package_folder_name>/azure-setup/scripts folder.
• Set execution permission to the Infrastructure deployment scripts.
  ‎
  • cd <package_folder_name>/azure-setup/scripts
  • chmod +x *.sh
    ‎
• Execute the shell script deploy_base_infra.sh in Azure cloud shell by passing required parameters

Parameters

Base Infrastructure Deployment validation

Select Deployments in respective resource group and look for ‘basedeployment’ to check the status of base infrastructure deployment.

Install Product Infrastructure

Once the package is downloaded, cd into <package_folder_name>/azure-setup/scripts folder.

Execute the shell script deploy_product_infra.sh in Azure cloud shell by passing required parameters

Parameters

Product Infrastructure Deployment validation

Select Deployments in respective resource group and look for ‘basedeployment’ to check the status of base infrastructure deployment.

Disk Encryption and Data Encryption

Data at rest is encrypted using encrypting Azure Disks using Disk encryption sets. The Data stored in Azure resources like Azure MySQL and Azure storage account is encrypted using keys stored in Azure Key Vault.

Azure Disk Encryption in AKS

‎  
‎Reference: https://docs.microsoft.com/en-us/azure/aks/azure-disk-customer-managed-keys 
‎ 
‎# Create a DiskEncryptionSet 
‎# key vault name, rg, etc needs to be changed accordingly 
‎#key key-smartops-k8s-disk-enc-001 ( key name given as an example ) needs to be created in Azure key Vault before creating the Disk Encryption Set  
‎  
‎keyVaultId=$(az keyvault show --name kv-engg-resrch-001 --query [id] -o tsv) 

keyVaultKeyUrl=$(az keyvault key show --vault-name kv-engg-resrch-001 --name key-smartops-k8s-disk-enc-001 --query [key.kid] -o tsv) 

az disk-encryption-set create -n smartops-k8s-des-001 -l eastus -g rg-smartopsengg-dev-001 --source-vault $keyVaultId --key-url $keyVaultKeyUrl 

 
‎Azure cloud shell:  
‎

 
‎  
‎  

Ensure Get, Wrap and Unwrap permission is set for the disk encryption set to the key created in Az key vault. 

Please refer des-itops-stg01 in above pic 

 
‎IMPORTANT: After creating the disk encryption set, select the disk encryption set and click on allow access to disk encryption key created in the key vault.  PFB pic 

K8s storage class 
‎#currently kept as a part of env-setup template. Can be changed as required  

#diskEncryptionSetID values needs to be changed accordingly (  subscriptions, resourceGroups, diskEncryptionSets)  

kind: StorageClass 

apiVersion: storage.k8s.io/v1 

metadata: 

  name: pvc-ade-custom-storage-class 

provisioner: kubernetes.io/azure-disk 

parameters: 

  kind: Managed 

  skuname: Premium_LRS 

  diskEncryptionSetID: "/subscriptions/dfaa090f-c407-4e75-ac08-143cb932bdcf/resourceGroups/rg-smartopsengg-dev-001/providers/Microsoft.Compute/diskEncryptionSets/smartops-k8s-des-001" 

 
‎ After deploying storage class, respective changes need to be made in statefulset’s pvcs referring to above custom storage class.

Data Encryption of Azure Database for MySQL with a customer-managed key 

References:  
‎ https://docs.microsoft.com/en-us/azure/mysql/howto-data-encryption-portal 

https://docs.microsoft.com/en-us/azure/mysql/concepts-data-encryption-mysql 

• Data Encryption of Azure MySQL instance is done at server level using CMK ( customer-managed key) 
• Key Terminologies  

Key Encryption Key [ KEK ] 

• Stored in Azure Key Vault 
• KEK is used to encrypt Data Encryption Key 

Data Encryption Key [ DEK ] 

Symmetric key used to encrypt a block of data 

• DEK, encrypted with KEK are stored separately 
• MySQL server needs below permissions on Azure Key Vault 

• get 
• wrapKey 
• UnwrapKey 

• Key Vault and Azure Database for MySQL must belong to same Azure AD 
• Enable soft delete feature on Key Vault instance 
• Key must be in ‘Enabled’ state 
• When keys are imported , only .pfx, .byok, .backup file formats are supported  
• If key vault generates the key , create a key backup before using for the first time.  

• Backup Azure Key Vault Key Reference 

When you configure data encryption with a customer-managed key in Key Vault, continuous access to this key is required for the server to stay online. If the server loses access to the customer-managed key in Key Vault, the server begins denying all connections within 10 minutes. The server issues a corresponding error message and changes the server state to Inaccessible. Some of the reason why the server can reach this state are: 
‎  

• If we create a Point in Time Restore server for your Azure Database for MySQL, which has data encryption enabled, the newly created server will be in Inaccessible state. You can fix this through Azure portal or CLI. 
• If we create a read replica for your Azure Database for MySQL, which has data encryption enabled, the replica server will be in Inaccessible state. You can fix this through Azure portal or CLI. 
• If you delete the Key Vault, the Azure Database for MySQL will be unable to access the key and will move to Inaccessible state. Recover the Key Vault and revalidate the data encryption to make the server Available. 
• If we delete the key from the Key Vault, the Azure Database for MySQL will be unable to access the key and will move to Inaccessible state. Recover the Key and revalidate the data encryption to make the server Available. 
• If the key stored in the Azure Key Vault expires, the key will become invalid and the Azure Database for MySQL will transition into Inaccessible state. Extend the key expiry date using CLI and then revalidate the data encryption to make the server Available. 

Limitations 

• Support for this functionality is limited to General Purpose and Memory Optimized pricing tiers. 
• This feature is only supported in regions and servers which support storage up to 16TB. For the list of Azure regions supporting storage up to 16TB, refer to the storage section in documentation here  
• Encryption is only supported with RSA 2048 cryptographic key. 

Steps 

• Create Azure Database for MySQL instance. PFB a sample screenshot for the confgurations.  

• Click Data Encryption on the ‘Security’ section and click ‘Yes’ for ‘Use customer managed key’ 

 
‎  

• After clicking ‘Yes’ for CMK. Select Key Vault and Key created in Azure Key Vault instance 

• Click Save 

• Restart Azure Database for MySQL 

 
‎Errors Observed while configuring  

If soft –delete is not enabled for keyvault , will get error like below  

• The above issue has been resolved when new keyvault instance created with soft delete enabled and enabling purge protection after key vault creation. 
  ‎   
• PFB screenshot after configuring CMK for enabling Data Encryption.   

Data Encryption of Azure Storage account using Customer Managed Key
‎  

Steps 
‎  

• Select the storage account from respective resource group which needs to be encrypted and click on encryption 

• Click ‘Customer-managed keys’ radio button  

Select the key vault and key by clicking ‘Select a key vault and key’  

• Select the key vault from the drop down list 
• Select the key from the key vault.  

• After selecting the key vault and key from the drop down list, click ‘Select’ button.  

• Click ‘Save’ to save the changes.  

Post Deployment Validations for Infrastructure

After successful deployments of base and product infrastructure using the shared ARM templates in the package, below validations needs to be performed before proceeding with the Application deployment.

Az MYSQL Firewall policies 
‎

• Public network access should be denied. 
• Allow access to Azure MySQL by configuring firewall policies 
• Enforce SSL connection should be Enabled 

Az MYSQL Max Connection settings

Max connection value for Azure MySQL is 300 by default. This needs to be set to 500.

Key vault permissions for the Azure AD user
‎
‎  

• Access to key vault can be enabled by configuring Key Vault access policies. 
• Before proceeding with deployment, we need to validate  

• Access is enabled for Disk Encryption 
• Permission model set as Vault access policy 
• Respective applications and resources are added with required access under APPLICATION section  

Azure service principal needs Get permission to Key Vault secrets. 

Disk Encryption set, storage account, Azure MySQL instance’s Key permissions. 
‎

Disk Encryption Set, Storage Account and Azure MySQL instance needs Get, Wrap and Unwrap key permissions.

Please refer below example for disk encryption set.  

 
‎

User’s access to Azure Key Vault instance

User’s access can be enabled by Adding users with required set of permission to access key, secret and certificate which is listed under USER section.

Please refer below screenshot.

Private Endpoint’s IP associations 
‎

• we have 5 private endpoints per environment. For these 5 private endpoints, we have 4 private DNZ zones. One private zone per Azure blob, Azure Key Vault , Azure MySQL and Azure Cache Redis instances.  
• There should be a private IP against each of these services. If this private IP is not associated to the respective private endpoints, application deployments will fail as the K8s cluster will not be able to communicate with these private endpoints.  We must manually add it via Azure portal, if found not associated.  
  ‎
• Finding IP address of a private endpoint – Please refer FAQ
  ‎  
• PFB screenshots for reference 

Key Vault Private Endpoint and its Private IP 
‎

Above Key Vault’s Private IP associated with respective Private Link of Key Vault 
‎

Update Private IP in Private Link  
‎

Please refer below example from product invoiceext to update the Private link when it is missing. Please check for required product itops accordingly.

• Please refer below screenshot when private endpoint’s IP is not associated with the private link.  
  ‎  
• When private endpoint IPs are not listed, include the same by adding record set of respective resource.  
  ‎

• When adding new recordset ,  update the instance name in Name field and add the private endpoint IP in the IP Address field.  

• If the recordset is present but the IP is not associated then Click the record set (Here Azure Key Vault kv-invoiceext-re01)  

• Update the Private IP in the ‘IP address’ field  

Node pools’ zone redundancy  

• Note: GPU pools does not have zone redundancy enabled since it is not supported in AKS.  

Kubernetes cluster connectivity
‎

After deploying all Azure resources for base and product infrastructure and before triggering the application deployment script, validate connectivity from Bastion VM to ITOps Kubernetes cluster.

Please refer Appendix

FTP Server Creation for ITOps

‎ Please use below attached script after substituting with required names for vnet , resource group etc.

APPENDIX

1. 1. ITOps Azure Resources Asset List
   2. ITOps Azure Resources Details
      • Base Infra
      • Product Infra
   3. Azure Disk Encryption
   4. SmartRecovery SOP
   5. ADFS Integration
   6. Data Archival & Restore documentation
   7. Connect to ITOps Kubernetes cluster from Bastion VM.

ITOps Azure Resources Asset List

ITOps Azure Resources Details

Base Infra

SmartOps VNET Address Space

Product Infra

AKS Node Pool Specification

Additional Resources

Other than base and product infra, we have the requirement of two isolated VMs.

1. 1. TCP Proxy VM for IHubLite
   2. FTP server

SmartOps DE team manages both the above VMs in maintenance subnet. At times when the client requirement comes in, then the required VMs has to be created in client subnet.

SmartRecovery Documentation

‎ SharePoint Location

ADFS Integration
‎

SharePoint Location

Connect to ITOps Kubernetes cluster from Bastion VM
‎

Connect to cluster (Kube config configured)  

• Select the Kubernetes cluster  
  ‎  
• Click ‘connect’ and follow the command provided in Azure portal 

‎ Please find below screenshots of the commands for reference on how to connect the Kubernetes cluster from Bastion VM.  These commands need to be executed from the Bastion VM.
‎

Known Issues during Infrastructure deployment and Resolutions.

Issues

Resolutions

Private Endpoint IP not associated with private link.
‎

Please refer Private Endpoint’s IP Associations

Azure ARM template deployment failures with status ‘Operation Timed Out’.
‎

This can occur intermittently because of latency or interruptions with Azure APIs. There are no specific fixes to be applied by Deployment engineer than executing the deployment script  again.
‎

Azure ARM template deployment with status ‘CONFLICT’.
‎

‘CONFLICT’ during deployments are observed only for NSGs & Subnets. There are no specific fixes to be applied by Deployment engineer than executing the deployment script again.
‎

File copying issues to Storage account blob containers
‎

Kubernetes cluster not able to access key vault
‎

Please refer Azure service principal needs Get permission to Key Vault secrets.

Disk Encryption set not able to get the keys
‎

Please refer Disk Encryption set, storage account, Azure MySQL instance’s Key permissions.
‎

Deployment Engineer not able to access Azure Key vault
‎

Please refer User’s access to Azure Key Vault instance
‎

Bad Request Status for Storage Account Resources.

Issue

Fix

• Access the VNET of the resource group and select Subnet. (Here Subnet of SmartOps Platform)
• Select the service endpoint ‘Microsoft.Storage’ in respective Subnet and click ‘Save’.

7.1 Known Limitations

Services which are not HA ready

Services which does not have Kubernetes health checks configured

FAQ

1. 1. What is the pre-requisite setup to be done before starting the Infrastructure deployment script?

      Please refer Prerequisite section of this doc.

   2. How to set the Domain Name when deploying base infrastructure?

      Domain name parameter can be left blank when executing the deploy_base_infra.sh if we need to create the domain name which have the environment name within it. This is taken care by the ARM template.

      E.g. if the environment name is dev01, and the domain name parameter name is left blank, ARM template will create the domain as smartops-dev01.eastus.cloudapp.azure.com

      It is possible to set custom domains. The custom domain can be passed as a parameter to deploy_base_infra.sh script.

       

   3. How to create an Azure Resource Group?

Login to Azure Portal and search ‘Resource Groups’
‎

Click ‘Add’

Select the correct Subscription from the drop-down list and provide a Resource group name and select the Region. Then click ‘Review + Create’

1. 1. What is the Disk capacity of AKS node pool?

‎Please refer AKS Node Pool Specifications.

1. 1. How to find IP address of Private Endpoint?
      ‎

Select the private endpoint of Azure resource

Click on Network Interface of Private Endpoint

IP address of private endpoint can be found out from the NIC of resource’s Private Endpoint.
‎

1. 1. How to create Azure MySQL password?
      ‎

Azure MySQL password is externalised and can be passed as a parameter to script deploy_product_infra.sh. (Please refer) One important point to take is to generate a strong password with alphanumeric characters.

1. 1. How to set max connection parameter of Azure MySQL?

Please refer