FinOps v7.1-Infrastructure Deployment Process

Important Terms
‎

 

Overview

‎Welcome to Kubernetes Infrastructure deployment guide for PWF IE 7.1. This document provides an overview of the flow of activities and overall architecture for installation of PWF IE 7.1
The main activities involved in this Kubernetes Infrastructure deployment guide is as below. 

• Base Infrastructure Installation in Azure Cloud

• Installation and Configuration of Kubernetes Infrastructure for PWF Invoice Extraction 

To get a deeper understanding about the installation read further about SmartOps 7.1 architecture. 

‎Architecture

PWF IE 7.1 is a distributed cloud application. It can be installed only in cloud infrastructure.  PWF IE 7.1 is deployed and managed in Kubernetes. Even though it is there in our roadmap to be cloud agnostic, at this point of time the installation is compatible and tested only on Microsoft Azure Cloud Infrastructure. Support for more cloud will follow in subsequent releases. 
The below diagram provides an overview 

 

• The installation is done on a single virtual network with multiple Subnets.  

• Each Subnets will be self-isolated with NSG rules and only required access granted. Each PWF and Platform will reside in its own Subnet.  

• Additionally, there will be a maintenance subnet where a bastion is hosted for administrative activities. 

• In certain situations, clients of SmartOps PWFs will have to get TCP connectivity to assets in SmartOps. For e.g. Posting data to a SmartOps Queue. In such cases as the diagram shows a VPN connectivity is established from client network to SmartOps Network. The connectivity is terminated at a DMZ subnet where a squid machine will isolate client network from SmartOps application subnets, for enhanced security. 

Prerequisites set up before downloading the package and deploying the infrastructure

Prerequisites
Azure CLI	Azure cli needs to be installed in Bastion VM if the infrastructure deployment scripts are executed from Bastion VM
Resource Group	An Azure resource group should be created before executing the infrastructure deployment scripts and needs to be passed as a parameter to the infrastructure deployment script.
Service Principal	Azure Service Principal client-id and client-secret is required to be passed as parameters to the infrastructure deployment script
Azure MYSQL Password	A strong password needs to be generated for Azure MYSQL and needs to be passed as a parameter to product infra deployment script.
Az Login	Before Downloading the package, we have to Az login using Azure AD account

 

Download the installation package in cloud shell

The Release packages are stored in SharePoint location and in azure artifacts. Please follow below steps for downloading the Azure Infrastructure installation scripts to Azure cloud shell.

 

Open Azure cloud shell

 

‎

az login in cloud shell

 

 

The package downloaded from SharePoint has to be made available in Azure cloud shell
‎

SharePoint location

https://ustglobal.sharepoint.com/teams/InnovationEngineering/Shared%20Documents/Forms/AllItems.aspx?viewid=f349a736%2D8a62%2D467f%2D8448%2D067be464bd59&id=%2Fteams%2FInnovationEngineering%2FShared%20Documents%2FKnowledge%20Management%2FSmartOps%20Deployment 

 

After downloading the package in cloud shell scripts for deploying

 

 

Scripts location for Base Infra and Product Infra deployment

 

After package is downloaded, the scripts for installing base and product infra will be available in

<package_folder_name>/azure-setup/scripts folder

 

 

 

ARM Templates for Base Infra and Product Infra deployment

 

After package is downloaded, the scripts for installing base and product infra will be available in

<package_folder_name>/azure-setup/arm-templates folder.

 

Install Base Infrastructure

• Once the package is downloaded, cd into <package_folder_name>/azure-setup/scripts folder.
• Set execution permission to the Infrastructure deployment scripts.
  ‎
  • cd <package_folder_name>/azure-setup/scripts
  • chmod +x *.sh
    ‎
• Execute the shell script deploy_base_infra.sh in Azure cloud shell by passing required parameters

Parameters

‎ Base Infrastructure Deployment validation

 

Select Deployments in respective resource group and look for ‘basedeployment’ to check the status of base infrastructure deployment.

 

 

Install Product Infrastructure

 

Once the package is downloaded, cd into <package_folder_name>/azure-setup/scripts folder.

 

Execute the shell script deploy_product_infra.sh in Azure cloud shell by passing required parameters

 

 

Parameters

 

Product Infrastructure Deployment validation

 

Select Deployments in respective resource group and look for ‘basedeployment’ to check the status of base infrastructure deployment.

 

Disk Encryption and Data Encryption

 

Data at rest is encrypted using encrypting Azure Disks using Disk encryption sets. The Data stored in Azure resources like Azure MySQL and Azure storage account is encrypted using keys stored in Azure Key Vault.

 

Azure Disk Encryption in AKS

‎  
‎Reference: https://docs.microsoft.com/en-us/azure/aks/azure-disk-customer-managed-keys 
‎ 
‎# Create a DiskEncryptionSet 
‎# key vault name, rg, etc needs to be changed accordingly 
‎#key key-smartops-k8s-disk-enc-001 ( key name given as an example ) needs to be created in Azure key Vault before creating the Disk Encryption Set  
‎  
‎keyVaultId=$(az keyvault show --name kv-engg-resrch-001 --query [id] -o tsv) 

keyVaultKeyUrl=$(az keyvault key show --vault-name kv-engg-resrch-001 --name key-smartops-k8s-disk-enc-001 --query [key.kid] -o tsv) 

az disk-encryption-set create -n smartops-k8s-des-001 -l eastus -g rg-smartopsengg-dev-001 --source-vault $keyVaultId --key-url $keyVaultKeyUrl 

 

 
‎Azure cloud shell:  
‎

 
‎  
‎  

Ensure Get, Wrap and Unwrap permission is set for the disk encryption set to the key created in Az key vault. 

 

 

 

 

 

 

Please refer des-platform-qa01 in above pic 

 
‎IMPORTANT: After creating the disk encryption set, select the disk encryption set and click on allow access to disk encryption key created in the key vault.  PFB pic 

 

 

K8s storage class 
‎#currently kept as a part of env-setup template. Can be changed as required  

#diskEncryptionSetID values needs to be changed accordingly (  subscriptions, resourceGroups, diskEncryptionSets)  

kind: StorageClass 

apiVersion: storage.k8s.io/v1 

metadata: 

  name: pvc-ade-custom-storage-class 

provisioner: kubernetes.io/azure-disk 

parameters: 

  kind: Managed 

  skuname: Premium_LRS 

  diskEncryptionSetID: "/subscriptions/dfaa090f-c407-4e75-ac08-143cb932bdcf/resourceGroups/rg-smartopsengg-dev-001/providers/Microsoft.Compute/diskEncryptionSets/smartops-k8s-des-001" 

 
‎ After deploying storage class, respective changes need to be made in statefulset’s pvcs referring to above custom storage class.

‎ Data Encryption of Azure Database for MySQL with a customer-managed key 

 

References:  
‎ https://docs.microsoft.com/en-us/azure/mysql/howto-data-encryption-portal 

https://docs.microsoft.com/en-us/azure/mysql/concepts-data-encryption-mysql 

 

• Data Encryption of Azure MySQL instance is done at server level using CMK ( customer-managed key) 
• Key Terminologies  

Key Encryption Key [ KEK ] 

• Stored in Azure Key Vault 
• KEK is used to encrypt Data Encryption Key 

Data Encryption Key [ DEK ] 

Symmetric key used to encrypt a block of data 

• DEK, encrypted with KEK are stored separately 
• MySQL server needs below permissions on Azure Key Vault 

• get 
• wrapKey 
• UnwrapKey 

• Key Vault and Azure Database for MySQL must belong to same Azure AD 
• Enable soft delete feature on Key Vault instance 
• Key must be in ‘Enabled’ state 
• When keys are imported , only .pfx, .byok, .backup file formats are supported  
• If key vault generates the key , create a key backup before using for the first time.  

• Backup Azure Key Vault Key Reference 

 

When you configure data encryption with a customer-managed key in Key Vault, continuous access to this key is required for the server to stay online. If the server loses access to the customer-managed key in Key Vault, the server begins denying all connections within 10 minutes. The server issues a corresponding error message and changes the server state to Inaccessible. Some of the reason why the server can reach this state are: 
‎  

• If we create a Point in Time Restore server for your Azure Database for MySQL, which has data encryption enabled, the newly created server will be in Inaccessible state. You can fix this through Azure portal or CLI. 
• If we create a read replica for your Azure Database for MySQL, which has data encryption enabled, the replica server will be in Inaccessible state. You can fix this through Azure portal or CLI. 
• If you delete the Key Vault, the Azure Database for MySQL will be unable to access the key and will move to Inaccessible state. Recover the Key Vault and revalidate the data encryption to make the server Available. 
• If we delete the key from the Key Vault, the Azure Database for MySQL will be unable to access the key and will move to Inaccessible state. Recover the Key and revalidate the data encryption to make the server Available. 
• If the key stored in the Azure Key Vault expires, the key will become invalid and the Azure Database for MySQL will transition into Inaccessible state. Extend the key expiry date using CLI and then revalidate the data encryption to make the server Available. 

  

Limitations 

• Support for this functionality is limited to General Purpose and Memory Optimized pricing tiers. 
• This feature is only supported in regions and servers which support storage up to 16TB. For the list of Azure regions supporting storage up to 16TB, refer to the storage section in documentation here  
• Encryption is only supported with RSA 2048 cryptographic key. 

 

Steps 

• Create Azure Database for MySQL instance. PFB a sample screenshot for the confgurations.  

 

• Click Data Encryption on the ‘Security’ section and click ‘Yes’ for ‘Use customer managed key’ 

 
‎  

• After clicking ‘Yes’ for CMK. Select Key Vault and Key created in Azure Key Vault instance 

 

 

• Click Save 

 

 

 

 

 

• Restart Azure Database for MySQL 

 

 
‎Errors Observed while configuring  

If soft –delete is not enabled for keyvault , will get error like below  

 

 

• The above issue has been resolved when new keyvault instance created with soft delete enabled and enabling purge protection after key vault creation. 
  ‎   
• PFB screenshot after configuring CMK for enabling Data Encryption.   

 

 

 

 

Data Encryption of Azure Storage account using Customer Managed Key
‎  

Steps 
‎  

• Select the storage account from respective resource group which needs to be encrypted and click on encryption 

 

 

 

• Click ‘Customer-managed keys’ radio button  

 

 

 

 

 

 

Select the key vault and key by clicking ‘Select a key vault and key’  

• Select the key vault from the drop down list 
• Select the key from the key vault.  

 

• After selecting the key vault and key from the drop down list, click ‘Select’ button.  

 

 

 

 

 

 

• Click ‘Save’ to save the changes.  

 

 

 

 

 

Post Deployment Validations for Infrastructure

 

After successful deployments of base and product infrastructure using the shared ARM templates in the package, below validations needs to be performed before proceeding with the Application deployment.

 

Az MYSQL Firewall policies 
‎

 

• Public network access should be denied. 
• Allow access to Azure MySQL by configuring firewall policies 
• Enforce SSL connection should be Enabled 

 

 

 

 

 

‎  

Az MYSQL Max Connection settings

Max connection value for Azure MySQL is 300 by default. This needs to be set to 500.

 

‎
‎
‎
‎Key vault permissions for the Azure AD user
‎
‎  
‎
‎

• Access to key vault can be enabled by configuring Key Vault access policies. 
• Before proceeding with deployment, we need to validate  

• Access is enabled for Disk Encryption 
• Permission model set as Vault access policy 
• Respective applications and resources are added with required access under APPLICATION section  

 

 

 

 

Azure service principal needs Get permission to Key Vault secrets. 

 

Disk Encryption set, storage account, Azure MySQL instance’s Key permissions. 
‎

Disk Encryption Set, Storage Account and Azure MySQL instance needs Get, Wrap and Unwrap key permissions.

Please refer below example for disk encryption set.  

 

 
‎

User’s access to Azure Key Vault instance

 

User’s access can be enabled by Adding users with required set of permission to access key, secret and certificate which is listed under USER section .

Please refer below screenshot.

 

 

Private Endpoint’s IP associations 
‎

• we have 4 private endpoints per environment. For these 4 private endpoints, we have 3 private DNZ zones. One private zone per Azure blob, Azure Key Vault and Azure MySQL instances.  
• There should be a private IP against each of these services. If this private IP is not associated to the respective private endpoints, application deployments will fail as the K8s cluster will not be able to communicate with these private endpoints.  We must manually add it via Azure portal, if found not associated.  
  ‎
• Finding IP address of a private endpoint – Please refer FAQ
  ‎  
• PFB screenshots for reference 

 

Key Vault Private Endpoint and its Private IP 
‎

 

  

Above Key Vault’s Private IP associated with respective Private Link of Key Vault 
‎

 

Update Private IP in Private Link  
‎

• Please refer below screenshot when private endpoint’s IP is not associated with the private link.  
  ‎  
• When private endpoint IPs are not listed, include the same by adding record set of respective resource.  
  ‎

 

• When adding new recordset ,  update the instance name in Name field and add the private endpoint IP in the IP Address field.  

 

• If the recordset is present but the IP is not associated then Click the record set (Here Azure Key Vault kv-invoiceext-re01)  

 

 

• Update the Private IP in the ‘IP address’ field  

 

 

 

 

 

 

 

 

Node pools’ zone redundancy  

 

• Note: GPU pools does not have zone redundancy enabled since it is not supported in AKS.  

 

Kubernetes cluster connectivity
‎

After deploying all Azure resources for base and product infrastructure and before triggering the application deployment script, validate connectivity from Bastion VM to Invoice Extraction Kubernetes cluster.

Please refer Appendix

‎ FTP Server Creation for Invoice Extraction

‎ Please use below attached script after substituting with required names for vnet , resource group etc.

APPENDIX

1. 1. Invoice Extraction Azure Resources Asset List
   2. Invoice Extraction Azure Resources Details
      • Base Infra
      • Product Infra
   3. Azure Disk Encryption
   4. SmartRecovery SOP
   5. ADFS Integration
   6. Data Archival & Restore documentation
   7. Connect to Invoice Extraction Kubernetes cluster from Bastion VM.

 

Invoice Extraction Azure Resources Asset List

 

Invoice Extraction Azure Resources Details

Base Infra

 

‎ SmartOps VNET Address Space

‎
‎ Product Infra

 

 

 

 

 

 

‎ AKS Node Pool Specification

 

Additional Resources

Other than base and product infra, we have the requirement of two isolated VMs.

1. 1. TCP Proxy VM for IHubLite
   2. FTP server

SmartOps DE team manages both the above VMs in maintenance subnet. At times when the client requirement comes in, then the required VMs must be created in client subnet.

SmartRecovery Documentation

‎ SharePoint Location

 

ADFS Integration
‎

SharePoint Location

Data Archival & Restore documentation
‎

Data Archival - SharePoint Location

Restore Checklist – SharePoint Location

 

Connect to Invoice Extraction Kubernetes cluster from Bastion VM
‎

Connect to cluster (Kube config configured)  

• Select the Kubernetes cluster  
  ‎  
• Click ‘connect’ and follow the command provided in Azure portal 

 

 

‎ Please find below screenshots of the commands for reference on how to connect the Kubernetes cluster from Bastion VM.  These commands needs to be executed from the Bastion VM.
‎

 

 

Known Issues during Infrastructure deployment and Resolutions.

Issues

 

 

 

Resolutions

 

Private Endpoint IP not associated with private link.
‎

Please refer Private Endpoint’s IP Associations

 

Azure ARM template deployment failures with status ‘Operation Timed Out’.
‎

This can occur intermittently because of latency or interruptions with Azure APIs. There are no specific fixes to be applied by Deployment engineer than executing the deployment script  again.
‎

Azure ARM template deployment with status ‘CONFLICT’.
‎

‘CONFLICT’ during deployments are observed only for NSGs & Subnets. There are no specific fixes to be applied by Deployment engineer than executing the deployment script again.
‎

File copying issues to Storage account blob containers
‎

 

Kubernetes cluster not able to access key vault
‎

Please refer Azure service principal needs Get permission to Key Vault secrets.

 

Disk Encryption set not able to get the keys
‎

Please refer Disk Encryption set, storage account, Azure MySQL instance’s Key permissions.
‎

Deployment Engineer not able to access Azure Key vault
‎

Please refer User’s access to Azure Key Vault instance
‎

Bad Request Status for Storage Account Resources.

Issue

 

Fix

• Access the VNET of the resource group and select Subnet. (Here Subnet of SmartOps Platform)
• Select the service endpoint ‘Microsoft.Storage’ in respective Subnet and click ‘Save’.

7.1 Known Limitations

Services which are not HA ready

 

 

 

 

 

 

 

Services which does not have Kubernetes health checks configured

 

FAQ

 

1. 1. What is the pre-requisite setup to be done before starting the Infrastructure deployment script?

      Please refer Prerequisite section of this doc.

   2. How to set the Domain Name when deploying base infrastructure?

      Domain name parameter can be left blank when executing the deploy_base_infra.sh if we need to create the domain name which have the environment name within it. This is taken care by the ARM template.

      E.g. if the environment name is dev01, and the domain name parameter name is left blank, ARM template will create the domain as smartops-dev01.eastus.cloudapp.azure.com

      It is possible to set custom domains. The custom domain can be passed as a parameter to deploy_base_infra.sh script.

       

   3. How to create an Azure Resource Group?

 

Login to Azure Portal and search ‘Resource Groups’
‎

Click ‘Add’

 

Select the correct Subscription from the drop-down list and provide a Resource group name and select the Region. Then click ‘Review + Create’

 

1. 1. What is the Disk capacity of AKS node pool?

‎Please refer AKS Node Pool Specifications.

 

1. 1. How to find IP address of Private Endpoint?
      ‎

Select the private endpoint of Azure resource

Click on Network Interface of Private Endpoint

IP address of private endpoint can be fond out from the NIC of resource’s Private Endpoint.
‎

 

1. 1. How to create Azure MySQL password?
      ‎

Azure MySQL password is externalised and can be passed as a parameter to script deploy_product_infra.sh. (Please refer) One important point to take care is to generate a strong password with alphanumeric characters.

 

1. 1. How to set max connection parameter of Azure MySQL?

Please refer