Smart Vision v2.5-Application Deployment Process

Important Terms  

 
‎Prerequisites   

• Base infra and infrastructure specific to Invoice Extraction available in the environment. Refer infrastructure creation document (WIP) to get an understanding on how to create the infrastructure. ‎ 

• Installation engineer has access to bastion VM (VM having visibility to the Invoice Extraction Kubernetes Cluster)   

• Bastion VM has ability to connect to K8s API server of invoice extraction (kubectl commands work.) 

• Access to the invoice extraction key vault is allowed for installation engineer and Bastion VM can connect to Key Vault (Configured via network firewall of Key vault) 

Download the installation package in bastion VM 

The Release packages are stored in SharePoint location and in azure artifacts. Please follow below steps for downloading. 

Primary Download location: Share point  

1. Navigate to sharepoint location:https://ustglobal.sharepoint.com/teams/InnovationEngineering/Shared%20Documents/Forms/AllItems.aspx?viewid=f349a736%2D8a62%2D467f%2D8448%2D067be464bd59&id=%2Fteams%2FInnovationEngineering%2FShared%20Documents%2FKnowledge%20Management%2FSmartOps%20Deployment  

2. Open the required release folder (eg:7.0.3) 

3. Download the product zip and move it to the target VM 

Secondary download location: Azure Artifacts  

Prerequisite: using below command install az cli in the target deployment vm   

1. Navigate to 
   https://dev.azure.com/USTInnovationEngineering/SmartOps/_packaging?_a=feed&feed=Smartops_Releases    

2. Click on the required package.  
   

3. Click on Versions  
    

4. Click on the options button (…) and select ‘Copy Install Command’. The download command will get copied to clipboard  
    

5. Login to VM where you want to extract the package and execute the command.
   

Note:  If its first time you will be prompted to install azure-devops extension. Give ‘Y’ and hit enter to continue.  

Validation of Product Infra before Application Install ‎  

Az MYSQL Firewall policies 

• Public network access should be denied. 

• Allow access to Azure MySQL by configuring firewall policies 

• Enforce SSL connection should be Enabled 
  ‎  

Key vault permissions for the Azure AD user. ‎ 
‎  

1. Access to key vault can be enabled by configuring Key Vault access policies. 

2. Before proceeding with deployment, we need to validate  

   1. Access is enabled for Disk Encryption 

   2. Permission model set as Vault access policy 

   3. Respective applications and resources are added with required access under APPLICATION section  

   4. Azure service principal needs Get permission to Key Vault secrets. 

• Disk Encryption set, storage account, Azure MySQL instance needs GET, Wrap and Unwrap Key permissions. Please refer below example for disk encryption set.  

   ‎

• User’s access can be enabled by Adding users with required set of permission to access key, secret and certificate which is listed under USER section  

Private Endpoint’s IP associations 

1. ‎we have 4 private endpoints per environment. For these 4 private endpoints, we have 3 private DNZ zones. One private zone per Azure blob, Azure Key Vault and Azure MySQL instances.  

2. There should be a private IP against each of these services. If this private IP is not associated to the respective private endpoints, application deployments will fail as the K8s cluster will not be able to communicate with these private endpoints.  We must manually add it via Azure portal, if found not associated.  

PFB screenshots for reference  

Key Vault Private Endpoint and its Private IP 
‎

 Above Key Vault’s Private IP associated with respective Private Link of Key Vault 
‎

Update Private IP in Private Link  
‎

• Please refer below screenshot when private endpoint’s IP is not associated with the private link.  
  ‎  

• When private endpoint IPs are not listed, include the same by adding record set of respective resource.  ‎

• When adding new record set ,  update the instance name in Name field and add the private endpoint IP in the IP Address field.  

• If the record set is present but the IP is not associated then Click the record set (Here Azure Key Vault kv-invoiceext-re01)  

• Update the Private IP in the ‘IP address’ field  

• Node pools’ zone redundancy  

• Note: GPU pools does not have zone redundancy enabled since it is not supported in AKS.  

Prepare the bastion VM  ‎  

Install Tools: script to install prerequisite packages in bastion VM 
‎

•  cd <package_path>/kubernetes/azure-setup/scripts ‎

  ‎

• Set execution permission  

  chmod +x installbastiontools.sh

• Execute Shell script 

  . / installbastiontools.sh

  Connect to cluster (Kube config configured)  

• Select the Kubernetes cluster  ‎  

• Click ‘connect’ and follow the command provided in Azure portal 

• Please find below screenshots of the commands for reference on how to connect the Kubernetes cluster from Bastion VM.   

Pre-Check condition:  Please check Python 3.6 is installed in the bastion VM. [ SmartInstall runs on Python 3.6] 

• Python 3.6 installation link reference 

• Ubuntu 18.04 comes with Python 3.6 as default.  

Application Install  

Setting secrets in Azure Key Vault.  

• Before starting the deployment, we must set secrets in Azure Key Vault using create-az-kv-secrets.sh script in kv-init folder with the updated values with respect to the product infrastructure created.  

• < Include kv init sc in package >  ‎  

• Before executing the create-az-kv-secrets.sh script, respective access policies must be set for the key vault for the Azure AD user.  ‎  

• Before executing the create-az-kv-secrets.sh script, Key Vault’s firewall policies should be allowed access from All networks. Please refer below screenshot. 

• < Note : As an alternate White list the IP through f/w policy or just allow all n/w > 

• Execute secret creation script against the Key Vault  

Azure Login from bastion VM 

Once Signed in successfully, there will be message in the browser like below ‎  

• Set the Subscription by below command

   az account set - -subscription <sunscription_id>  

   

• After updating the create-az-kv-secrets.sh with latest values with respect to the product infrastructure, execute the script with using below syntax 

• On Successful completion, all the required secrets will get be created in Azure Key Vault instance.  ‎

• Once the script is executed successfully and validated the secrets created, switch back Key Vault’s firewall policy from ‘All networks’ to ‘Private endpoint and selected networks’ and save the change. Please refer below screenshot.  

Key Vault Additions

Refer kv-init/create-az-kv-secrets-prod.sh to run it as a script.

1. <NAMESPACE>-azure-ocr-key  with value  "respective key value "

‎ Create / Update Env JSON file 

• Smart Install uses environment JSON file to install respective application.

• Before deployment, environment JSON file needs to be updated as required.

• A template of environment JSON is available in the package

• Please refer below environment JSON file key values and its details 

There are two paths for application install. You can take only one of the routes. 

1. Restore data from an old Environment (E.g., 6.4.3) and install Invoice Extraction 

2. Fresh Install Invoice Extraction.   

Option 1: Restore data from an old Environment and install Invoice Extraction 

Steps 

1. Make sure data is copied to backup storage account

• Allow all networks if source is outside of the network.

• Update env json file with back up path and file names ( datarestore section)  

• Please find below details of the key values which needs to be updated accordingly in the environment json file for restore deployments.  

• Please find below example of environment json file used for restoring databases. 

Once the packages are downloaded to bastion VM, execute below commands to Install the application with data restore 
‎  

cd <package_path>/kubernetes/ smartinstall 

• Open tool “screen” to have an uninterrupted session for running deployment

• Execute restore.py 

• product – The application which needs to be deployed e.g. invoiceext 

• env – The environment which the application needs to be deployed e.g dev, qa 

• The environment must be named after naming the product infrastructure.  

• The env name should be not more than 3.  

• kubecontext - The kubecontext of the product infrastructre 

If the process is disrupted and ended due to any error in Infrastructure, restart execution using same above command for restore after correcting infrastructure.
‎ If there is any interruption in network connectivity ,the session can be resumed using below command from bastion VM

If the process is disrupted and ended due to any error, restart execution using

• Validation of restore started successfully  

• Message from smartinstall when data restore started 

Verify the restore process has started successfully via K9s 

• Ensure smart-recovery-job container is in running state 

• Check the logs via K9s 

• Once the restore is completed verify following  

• Check for any errors in logs. 

Sample Error Log where the restore has failed.       

1.  MySQL - access the server through PHPMyAdmin and ensure all dbs are in place. Analyze the logs for list of MySQL databases and tables restored.  

   1. Mongo - access Mongo Express and verify  

   2. RabbitMQ - via RabbitMQ management UI 

   3. Elasticsearch - Analyse smart-recovery-job pod's logs and ensure the elasticsearch restore has completed successfully 

2. Minio - Verify Files in IE appfilestore buckets. 

Smart Recovery Start  

 MySQL Restore 

 MySQL restore completed 
‎  Mongo Restore  

 Mongo restore completed 

 Minio Restore 

 
Note: Minio restore completion can be verified through Kibana logs.

Elasticsearch Restore 
‎

Steps for Database Validation of Data counts after restore process

Verify Mysql Table Counts.

Following query can be used for Mysql validation to fetch table counts from current production environment.Execute the query by logging in existing production environment.

SELECT TABLE_SCHEMA ,table_name, table_rows FROM INFORMATION_SCHEMA.TABLES

The mysql table data information is printed in Smart-Recovery logs as in below screenshots in restored environement

Verify Minio Table counts

Login to azure portal and open backup storage account.(eg: sasinvoiceextbackupdrn01)

Open Storage expolorer and move to the back up container. Calculate the blob count in buckets available in minio-data-backup folder,by clickin gon folder statistics as in below screen shot

The blob count will be displayed as in below screen shot

Once restore process completed successfully for minio as mentioned in restore process completion, the file count for minio can be validated in storage account for App file Store.

Enable all networks for appfile store.

Validate the count of buckets restored from backup by clicking the folder properties

Set the Networking setting back to “Selected Networks” once the validation completes

• How to monitor the deployments. (k9s)  

• Once the restore is completed successfully, run smartinstall in install mode to deploy Invoice Extraction

• Open tool “screen” to have an uninterrupted session for running deployment

• 

  cd <package_path>/kubernetes/smartinstall

• Execute install.py 

• product – The application which needs to be deployed e.g. invoiceext 

• env – The environment which the application needs to be deployed e.g dev , qa 

• kubecontext - The kubecontext of the product infrastructre 

If there is any interruption in the connectivity ,the session can be resumed using below command

• Once installation of all stacks are completed, generate offline token and restart application stacks(Refer Appendix)

Option 2: Fresh Install Invoice Extraction 

 Once the packages are downloaded to bastion VM , execute below commands to Install the application without data restore 

Open tool “screen” to have an uninterrupted session for running deployment

• cd <package_path>/kubernetes/ smartinstall 

• Execute installWithDataInit.py 

• product – The application which needs to be deployed e.g. invoiceext 

• env – The environment which the application needs to be deployed e.g dev , qa 

• kubecontext - The kubecontext of the product infrastructure 

Open tool “screen” to have an uninterrupted session for running deployment

HTTPS Enablement after Application Installation

Please follow the steps mentioned in below video for Certificate generation(Internal to Smartops Team)

https://web.microsoftstream.com/video/fc814048-9405-423d-adca-22d28ecc30bc?list=trending

Once the certificate is available follow below steps to enable HTTPS enablement

Application gateway Settings to Upload certificates

1. Login to Azure portal

2. Select application gateway created for the environment

3. Select Listeners and add a new listener for https as in following screen shots

   

    

4. Upload the verified certificate file in .pfx format for the environment and provide the Cert-name same as the pfx name

5. Use the same password used for generating the certificates

    

   

6. Listener added successfully

   

7. Create a new rule for redirecting http traffic to https listener.

   

8. Select Back end Target values as below by selecting redirection and setting Target URL as “https://<dns-name>”

   

    

9. Edit existing rule “route-smartops-apps” and select the listener as the newly created “smartops-https-listener” as below

   

10. Also reset the targetURL in backend targets for path based rules as “https”

11. Click on Save to get the changes reflected.

Validation for Successful Deployment

After Successful completion of Smart install installations, access the deployment in k9s and check all pods are in ready state

All pods created via Kubernetes jobs will be in completed state

Access Key cloak admin URL(Refer Appendix) and maintenance URL s from windows VM to verify the URL access

After Validation Generate offline token and restart application Stacks

Appendix 

Appendix consists of the following sections 
‎

• Creating Keys in Azure Key Vault for Data Encryption 
• Azure Disk Encryption in AKS
• Data Encryption of Azure Database for MySQL with a customer-managed key 
• Data Encryption of Azure Storage account using Customer Managed Key 
• Secondary download location: Azure Artifacts 
• List of containers for which autoscaling is enabled 
• Clear existing Databases to rerun Restore process
• Generate offline token and restart application stacks
• Maintenance URLS
• AD Integration Steps
• Create Dev User for grafana
• Sample commands

Creating Keys in Azure Key Vault for Data Encryption 

For enabling Data Encryption for Azure MySQL, Storage Accounts and enabling Disk Encryption for Volumes in Kubernetes cluster, we need to create Encryption keys in Azure Key vault which is used to encrypt the data.  
‎  

• Select the key vault and click ‘Keys’ 

 
‎  

• Click ‘Generate/Import’ 

• Set a Key name and click ‘Create’. Key Type and Key Size can be set with the default values unless there is a specific requirement.  

Following are the additions to keyvaults as part of Smart Vision 2.4:

<NAMESPACE>-rhub-rsa-public-key with value TBA
<NAMESPACE>-rhub-rsa-private-key with value TBA
<NAMESPACE>-jfrog-artifactory-username with value so-maven-publisher
<NAMESPACE>-jfrog-artifactory-password with value @rt1factPublisher

Azure Disk Encryption in AKS

‎ Reference: https://docs.microsoft.com/en-us/azure/aks/azure-disk-customer-managed-keys 
‎ 
‎# Create a DiskEncryptionSet 
‎ # key vault name, rg, etc needs to be changed accordingly 
‎ #key key-smartops-k8s-disk-enc-001  ( key name given as an example ) needs to be created in Azure key Vault before creating the Disk Encryption Set  
‎  
‎keyVaultId=$(az keyvault show --name kv-engg-resrch-001 --query [id] -o tsv) 

keyVaultKeyUrl=$(az keyvault key show --vault-name kv-engg-resrch-001 --name key-smartops-k8s-disk-enc-001 --query [key.kid] -o tsv) 

az disk-encryption-set create -n smartops-k8s-des-001 -l eastus -g rg-smartopsengg-dev-001 --source-vault $keyVaultId --key-url $keyVaultKeyUrl 

 
‎ Azure cloud shell:  
‎

 
‎  

Ensure Get, Wrap and Unwrap permission is set for the disk encryption set to the key created in Az key vault. 

Please refer des-platform-qa01 in above pic 

 IMPORTANT: After creating the disk encryption set, select the disk encryption set and click on allow access to disk encryption key created in the key vault.   PFB pic 

K8s storage class 
‎ #currently kept as a part of env-setup template. Can be changed as required  

#diskEncryptionSetID values needs to be changed accordingly (  subscriptions, resourceGroups, diskEncryptionSets)  

kind: StorageClass 

apiVersion: storage.k8s.io/v1 

metadata: 

  name: pvc-ade-custom-storage-class 

provisioner: kubernetes.io/azure-disk 

parameters: 

  kind: Managed 

  skuname: Premium_LRS 

  diskEncryptionSetID: "/subscriptions/dfaa090f-c407-4e75-ac08-143cb932bdcf/resourceGroups/rg-smartopsengg-dev-001/providers/Microsoft.Compute/diskEncryptionSets/smartops-k8s-des-001" 

 
‎ After deploying storage class, respective changes need to be made in statefulset’s pvcs referring to above custom storage class.  
‎
‎

Data Encryption of Azure Database for MySQL with a customer-managed key 

References:  
‎ https://docs.microsoft.com/en-us/azure/mysql/howto-data-encryption-portal 

https://docs.microsoft.com/en-us/azure/mysql/concepts-data-encryption-mysql 

• Data Encryption of Azure MySQL instance is done at server level using CMK ( customer-managed key) 

• Key Terminologies  

Key Encryption Key [ KEK ] 

• Stored in Azure Key Vault 

• KEK is used to encrypt Data Encryption Key 

Data Encryption Key [ DEK ] 

Symmetric key used to encrypt a block of data 

• DEK, encrypted with KEK are stored separately 

• MySQL server needs below permissions on Azure Key Vault 

• get 

• wrapKey 

• UnwrapKey 

• Key Vault and Azure Database for MySQL must belong to same Azure AD 

• Enable soft delete feature on Key Vault instance 

• Key must be in ‘Enabled’ state 

• When keys are imported , only .pfx, .byok, .backup file formats are supported  

• If key vault generates the key , create a key backup before using for the first time.  

• Backup Azure Key Vault Key Reference 

When you configure data encryption with a customer-managed key in Key Vault, continuous access to this key is required for the server to stay online.  If the server loses access to the customer-managed key in Key Vault, the server begins denying all connections within 10 minutes. The server issues a corresponding error message and changes the server state to  Inaccessible . Some of the reason why the server can reach this state are: 
‎  

• If we create a Point in Time Restore server for your Azure Database for MySQL, which has data encryption enabled, the newly created server will be in  Inaccessible  state. You can fix this through  Azure portal  or  CLI. 

• If we create a read replica for your Azure Database for MySQL, which has data encryption enabled, the replica server will be in Inaccessible  state. You can fix this through  Azure portal or CLI. 

• If you delete the Key Vault, the Azure Database for MySQL will be unable to access the key and will move to Inaccessible  state. Recover the  Key Vault and revalidate the data encryption to make the server  Available . 

• If we delete the key from the Key Vault, the Azure Database for MySQL will be unable to access the key and will move to Inaccessible  state. Recover the  Key and revalidate the data encryption to make the server  Available . 

• If the key stored in the Azure Key Vault expires, the key will become invalid and the Azure Database for MySQL will transition into Inaccessible  state. Extend the key expiry date using  CLI and then revalidate the data encryption to make the server  Available . 

Limitations 

• Support for this functionality is limited to General Purpose and Memory Optimized pricing tiers. 

• This feature is only supported in regions and servers which support storage up to 16TB. For the list of Azure regions supporting storage up to 16TB, refer to the storage section in documentation  here 

• Encryption is only supported with RSA 2048 cryptographic key. 

Steps 

• Create Azure Database for MySQL instance. PFB a sample screenshot for the confgurations.  

• Click Data Encryption on the ‘Security’ section and click ‘Yes’ for ‘Use customer managed key’ 

 
‎  

• After clicking ‘Yes’ for CMK. Select Key Vault and Key created in Azure Key Vault instance 

• Click Save 

• Restart Azure Database for MySQL 

 
‎ Errors Observed while configuring  

If soft –delete is not enabled for keyvault , will get error like below  

• The above issue has been resolved when new keyvault instance created with soft delete enabled and enabling purge protection after key vault creation. 
  ‎   

• PFB screenshot after configuring CMK for enabling Data Encryption.   

Data Encryption of Azure Storage account using Customer Managed Key ‎ 

Steps 
‎  

• Select the storage account from respective resource group which needs to be encrypted and click on encryption 

• Click ‘Customer-managed keys’ radio button  

Select the key vault and key by clicking ‘Select a key vault and key’  

• Select the key vault from the drop down list 

• Select the key from the key vault.  

• After selecting the key vault and key from the drop down list, click ‘Select’ button.  

• Click ‘Save’ to save the changes.  

Secondary download location: Azure Artifacts  

1. Navigate to https://dev.azure.com/USTInnovationEngineering/SmartOps/_packaging?_a=feed&feed=Smartops_Releases  

1. Click on the required package.  
    
2. Click on Versions  
    
3. Click on the options button (…) and select ‘Copy Install Command’. The download command will get copied to clipboard  
    
4. Login to VM where you want to extract the package and execute the command. 

Please Note:  If its first time you will be prompted to install azure-devops extension. Give ‘Y’ and hit enter to continue. 

List of containers for which autoscaling is enabled  
‎

How to monitor Kubernetes deployments using K9s

K9s is installed when the installbastiontools.sh script is executed. Please refer

Staying in home directory execute below command to open K9s

K9s/k9s

Or

cd k9s

./k9s

Generating Offline Token

1. Login to keycloak master URL to generate offline token

   https://<dns-name>/paas/invoiceextraction/ieui/smartops/master/

    

   

2. Revoke existing offline Token

   

3. Confirm revoke process of existing offline token by clicking “Yes”

   

4. Click on “generate New offline Token” button

   

5. Confirm on generate new offline Token

   

6. Newly generated offline token will be available

   

7. Copy the offline token

8. Login to bastion VM

9. Do az login to with a user who has edit access for Secrets in Keyvault.

10. Execute below command to update the offlline token in Keyvault from bastionVM.

az keyvault secret set --subscription <subscription-id> --vault-name <keyvault-name> --name <product-name>-<env-name>-offline-token --value "offline-token-value" -e base64

eg:

az keyvault secret set --subscription cdf5c496-95b3-4219-9117-35d4e0746d13 --vault-name kv-invoiceext-drn01 --name invoiceextv1-drn01-offline-token --value "eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJhODdjYzgwOS02YTA1LTQyY2MtOTY3YS0zNjk3OGFjZGFkZTUifQ.eyJqdGkiOiIwNjcxZmU3Zi00NWYwLTQzZWUtOWEyMi0xZWFiN2EzMWUyOTEiLCJleHAiOjAsIm5iZiI6MCwiaWF0IjoxNjEwNTM4MzAzLCJpc3MiOiJodHRwczovL3NtYXJ0b3BzLWRybjAxLmVhc3R1cy5jbG91ZGFwcC5henVyZS5jb20vcGFhcy9pbnZvaWNlZXh0cmFjdGlvbi9rZXljbG9hay9hdXRoL3JlYWxtcy91c3RnbG9iYWwiLCJhdWQiOiJodHRwczovL3NtYXJ0b3BzLWRybjAxLmVhc3R1cy5jbG91ZGFwcC5henVyZS5jb20vcGFhcy9pbnZvaWNlZXh0cmFjdGlvbi9rZXljbG9hay9hdXRoL3JlYWxtcy91c3RnbG9iYWwiLCJzdWIiOiJiN2JhNTFlZS05MzFmLTQ3MDgtODNiNi1mZGRhMDc4OTUwNDEiLCJ0eXAiOiJPZmZsaW5lIiwiYXpwIjoic21hcnRvcHMtZnJvbnRlbmQiLCJhdXRoX3RpbWUiOjAsInNlc3Npb25fc3RhdGUiOiJiMmM1NzgxYi04YWQ0LTQ5NTUtYTBiZC05MWRlNTkxMTZhMTQiLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsib2ZmbGluZV9hY2Nlc3MiXX0sInNjb3BlIjoib2ZmbGluZV9hY2Nlc3MifQ._G6N72VU6X-N5IHHXDjzKHUjSWgglb4cKge6JNh93dc" -e base64

1. Open k9s/k9s

2. Recreate “vault-secret-sync” pod to get the updated offline token secret. Click on “Cntrl+d“ to recreate the pod

   

    

3. Following list of containers needs restart after offline token Updation

1. IN K9s/Deployments, select the respective deployments and scale down replicas to “0”

   

2. Wait for a minute and set back replica count to 2 again. The existing pods will be removed, and new pods will be created

1. Execute this step for all deployments in the list mentioned above

Maintenance URLS

Following set of URLs are used for maintenance purpose of application. Credentials for the maintenance URL is updated in below table.

Note: Maintenance URL access will be blocked from internet for production environments. Restriction rules are handled in WAF rules

Access to maintenance URLS are allowed from Windows bastion VM only.

Refer document for accessing production windows bastion VM

Eg:http://<ip_of_kub_internal_lb>/paas/<product>/kibana/

AD Integration Steps

Refer following Documentation for AD integration

AD Integration Document

*Contact SmartOps Support team< smartops-support-team@ust.com > for credentials to access online documentation

Known Issues

Post Deployment Activity from Deployment perspective

As we have changed the artifacts reference from Archiva to jFrog-Artifactory, it is expected to uninstall 2 stacks deployed for smartops-archiva

FAQ

1. When existing working offline token got invalidated after keycloak restartWhen existing working offline token got invalidated after keycloak restart

   1.    Login to SmartOps Master Admin UI, Revoke the offline tokens generated for all the organizations.
   2.    Navigate to Keycloak Administration console, do the below step(3) for all organizations except master.
   3.    Go to Users tab, select sense_master user, go to Consents tab, check if there are any Offline Token entries, if yes, then click on Revoke button. 
   
   4.    Uninstall keycloak services in K8.
   5.    Clear invalid offline token entries from DB, if any. Execute the below SQL scripts in MySQL.
   TRUNCATE table keycloak.offline_client_session;
   TRUNCATE table keycloak.offline_user_session;
   6.    Install keycloak services in K8.
   7.    Login to SmartOps Master Admin UI, Generate new offline token for all the organizations
   8.    Update keyvault with new token generated for USTGlobal
   9.    Restart secret vault sync pod to reflect the new value
   10.    Confirm that pods are using new offline token.