FinOps v7.0-Infrastructure Deployment Process

 

 

Important Terms
‎

Base Infra	The base Infrastructure which needs to be created before deploying product Infrastructure
Product Infra	The Kubernetes infrastructure required for respective product

 

 

Overview

Welcome to Kubernetes Infrastructure deployment guide for PWF IE 7.0. This document provides an overview of the flow of activities and overall architecture for installation of PWF IE 7.0

The main activities involved in this Kubernetes Infrastructure deployment guide is as below. 

1. Base Infrastructure Installation in Azure Cloud

2. Installation and Configuration of Kubernetes Infrastructure for PWF Invoice Extraction 

To get a deeper understanding about the installation read further about SmartOps 7.0 architecture. 

Architecture

PWF IE 7.0 is a distributed cloud application. It can be installed only in cloud infrastructure.  PWF IE 7.0 is deployed and managed in Kubernetes. Even though it is there in our roadmap to be cloud agnostic, at this point of time the installation is compatible and tested only on Microsoft Azure Cloud Infrastructure. Support for more cloud will follow in subsequent releases. 

The below diagram provides an overview 

 < We need insert the latest architecture diagram here > 

• The installation is done on a single virtual network with multiple Subnets.  

• Each Subnets will be self-isolated with NSG rules and only required access granted. Each PWF and Platform will reside in its own Subnet.  

• Additionally, there will be a maintenance subnet where a bastion is hosted for administrative activities. 

• In certain situations, clients of SmartOps PWFs will have to get TCP connectivity to assets in SmartOps. For e.g. Posting data to a SmartOps Queue. In such cases as the diagram shows a VPN connectivity is established from client network to SmartOps Network. The connectivity is terminated at a DMZ subnet where a squid machine will isolate client network from SmartOps application subnets, for enhanced security.‎

Download the installation package in cloud shell

The Release packages are stored in SharePoint location and in azure artifacts. Please follow below steps for downloading the Azure Infrastructure installation scripts to Azure cloud shell.

 

• Open Azure cloud shell

 

‎

• az login in cloud shell

 

 

• The package downloaded from SharePoint has to be made available in Azure cloud shell

  • SharePoint location

https://ustglobal.sharepoint.com/teams/InnovationEngineering/Shared%20Documents/Forms/AllItems.aspx?viewid=f349a736%2D8a62%2D467f%2D8448%2D067be464bd59&id=%2Fteams%2FInnovationEngineering%2FShared%20Documents%2FKnowledge%20Management%2FSmartOps%20Deployment 

• After downloading the package in cloud shell scripts for deploying

 

Scripts location for Base Infra and Product Infra deployment

After package is downloaded, the scripts for installing base and product infra will be available in

<package_folder_name>/azure-setup/scripts folder

 

 

ARM Templates for ase Infra and Product Infra deployment

After package is downloaded, the scripts for installing base and product infra will be available in

<package_folder_name>/azure-setup/arm-templates folder.

 

Install Base Infrastructure

• Once the package is downloaded, cd into <package_folder_name>/azure-setup/scripts folder.
  ‎

• Execute the shell script deploy_base_infra.sh in Azure cloud shell by passing required parameters

./deploy_base_infra.sh <subscription_id> <resource_group> <environment> <base_ip_seg> <smartops_domain_name> <sp_client_id> <tenant_id> <sp_client_secret>

 

Parameters

Base Infrastructure Deployment validation

Select Deployments in respective resource group and look for ‘basedeployment’ to check the status of base infrastructure deployment.

Install Product Infrastructure

• Once the package is downloaded, cd into <package_folder_name>/azure-setup/scripts folder.

• Execute the shell script deploy_product_infra.sh in Azure cloud shell by passing required parameters

./deploy_product_infra.sh <subscription_id> <resource_group> <environment> <product> <aksNodeImageVersion> <sp_client_id> <tenant_id> <sp_client_secret> <mysql_admin_password>

 

Parameters

 

Product Infrastructure Deployment validation

Select Deployments in respective resource group and look for ‘basedeployment’ to check the status of base infrastructure deployment.

 

Post Deployment Validations for Infrastructure

After successful deployments of base and product infrastructure using the shared ARM templates in the package, below validations needs to be performed before proceeding with the Application deployment.

 

Az MYSQL Firewall policies ‎

 

• Public network access should be denied. 
• Allow access to Azure MySQL by configuring firewall policies 
• Enforce SSL connection should be Enabled ‎ 

Key vault permissions for the Azure AD user. 
‎
‎ ‎‎

• Access to key vault can be enabled by configuring Key Vault access policies. 

• Before proceeding with deployment, we need to validate  

• Access is enabled for Disk Encryption 

• Permission model set as Vault access policy 

• Respective applications and resources are added with required access under APPLICATION section  

Azure service principal needs Get permission to Key Vault secrets. 

 

Disk Encryption set, storage account, Azure MySQL instance’s Key permissions‎

Disk Encryption Set, Storage Account and Azure MySQL instance needs Get, Wrap and Unwrap key permissions.

Please refer below example for disk encryption set.  

 

User’s access to Azure Key Vault instance

User’s access can be enabled by Adding users with required set of permission to access key, secret and certificate which is listed under USER section .

Please refer below screenshot.

 

Private Endpoint’s IP associations ‎

• we have 4 private endpoints per environment. For these 4 private endpoints, we have 3 private DNZ zones. One private zone per Azure blob, Azure Key Vault and Azure MySQL instances.  

• There should be a private IP against each of these services. If this private IP is not associated to the respective private endpoints, application deployments will fail as the K8s cluster will not be able to communicate with these private endpoints.  We must manually add it via Azure portal, if found not associated.  
  ‎  

• PFB screenshots for reference 

 

Key Vault Private Endpoint and its Private IP 
‎

 

Above Key Vault’s Private IP associated with respective Private Link of Key Vault 
‎

 

Update Private IP in Private Link  
‎

• Please refer below screenshot when private endpoint’s IP is not associated with the private link.  
  ‎  

• When private endpoint IPs are not listed, include the same by adding record set of respective resource.  
  ‎

 

• When adding new recordset ,  update the instance name in Name field and add the private endpoint IP in the IP Address field.   

• If the recordset is present but the IP is not associated then Click the record set (Here Azure Key Vault kv-invoiceext-re01)  

 

• Update the Private IP in the ‘IP address’ field  

 

Node pools’ zone redundancy  

 

Note: GPU pools does not have zone redundancy enabled since it is not supported in AKS.  
‎ FTP Server Creation for Invoice Extraction

‎Please use below attached script after substituting with required names for vnet , resource group etc.

 

APPENDIX

Invoice Extraction Azure Resources Asset List

 

Invoice Extraction Azure Resources Details

Base Infra

 

 

Product Infra

 

 

 

Known Issues during Infrastructure deployment and Resolutions.

Issues

 

Resolutions

 

Private Endpoint IP of Azure MySQL, Storage accounts and key vault not associated with respective private link.
‎

Please refer Private Endpoint’s IP Associations

 

Azure ARM template deployment failures with status ‘Operation Timed Out’. This can be resolved by redeploying the template.‎

This can occur intermittently because of latency or interruptions with Azure APIs. There are no specific fixes to be applied by Deployment engineer than executing the deployment script  again.
‎

Azure ARM template deployment with status ‘CONFLICT’. This can be resolved by redeploying the  template.
‎

‘CONFLICT’ during deployments are observed only for NSGs. There are no specific fixes to be applied by Deployment engineer than executing the deployment script again.
‎

File copying issues to Storage account blob containers because of firewall policy restricted to ‘all networks.

 

 

Kubernetes cluster not able to access key vault because of not setting correct access policies

Please refer Azure service principal needs Get permission to Key Vault secrets.

 

Disk Encryption set not able to get the keys because of not having the Get Wrap and Unwrap permission to key stored in Azure Key Vault.
‎

Please refer Disk Encryption set, storage account, Azure MySQL instance’s Key permissions.
‎

Deployment Engineer not able to access Azure Key vault because of not having required access policies set in Azure Key Vault.
‎

Please refer User’s access to Azure Key Vault instance
‎

Bad Request Status for Storage Account Resources.

Issue

 

 

Fix

• Access the VNET of the resource group and select Subnet. (Here Subnet of SmartOps Platform)

• Select the service endpoint ‘Microsoft.Storage’ in respective Subnet and click ‘Save’.

 

 

FAQ