Azure CLI: Determining location of CA certs to work with Azure Stack Hub/ASDK

I’ve been doing some work on Azure Stack HUB (ASH) and ASDK recently, and the perennial problem with certificates has raised it’s head again. This is a quick blog post for anyone using Linux and Azure CLI to administer to figure out where you should store the CA root certificates, as the documentation is somewhat vague.

• First, make sure you install the latest version of Az cli and not version 2.29, as that version has CVE’s (versions prior to 2.40.0 are affected)

• Once installed, check the version and what Python version is used (We need to make sure that any Python commands we are running uses this version. )

az --version

View fullsize

• Next, install pip for the python version the az cli is using (in this case it’s Python 3.9. but future versions could change)

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python3.9 get-pip.py

• Install the Certifi module:

pip install certifi

• Now you can determine where the cacert.pem file is located as used by az cli:

python3.9 -c "import certifi; print(certifi.where())"

• Now you can add the ASH or ASDK CA certs to the store for use by Azure CLI:

cat <CA Cert>.pem >> ~/.local/lib/python3.9/site-packages/certifi/cacert.pem

You can use the docs here to obtain the CA root certificate, or if you’re running a Linux VM from within ASH/ASDK, simply run:

sudo cat /var/lib/waagent/Certificates.pem >> ~/.local/lib/python3.9/site-packages/certifi/cacert.pem

- If you were to follow the Microsoft docs, I found az cli would still not be able to communicate successfully.

It is necessary to run the following:

export REQUESTS_CA_BUNDLE=~/.local/lib/python3.9/site-packages/certifi/cacert.pem
# RECOMMENDED: set the env var automatically for your subsequent sessions
echo 'export REQUESTS_CA_BUNDLE=~/.local/lib/python3.9/site-packages/certifi/cacert.pem' >> ~/.bash_profile

As you can see above, I have been able to run az cli targeting ASDK, whereas before, it would throw the SSL error.

Tested on CentOS 8 and Rocky Linux 8.5

Deploying ASDK 2206 to an Azure VM

Azure Stack Hub version 2206 was release a couple of months ago, but anyone trying to deploy ASDK 2206 to Azure will have found that the latest version is 2108. Until the official method is updated, here’s how you can do it .

I’m using the awesome scripts by Yagmur Sahin as the basis for the solution: https://github.com/yagmurs/AzureStack-VM-PoC

• Open the Azure portal and then create a PowerShell Cloud Shell.

View fullsize

I recommend you reset the user settings, as there can be issues with versions of the Azure PowerShell modules.

• Run the following command in the new PowerShell session:

git clone https://github.com/dmc-tech/AzureStack-VM-PoC.git

• Run the following, changing to meet your requirements. VirtualMachineSize can be from the following sizes:

  "Standard_E32s_v3",

  "Standard_E48s_v3"

cd ./AzureStack-VM-PoC/ARMv2

$ResourceGroupName = 'asdk01-uks'
$Region = 'uk south' 
$VirtualMachineSize = 'Standard_E48s_v3' 
$DataDiskCount = 11

./Deploy-AzureStackonAzureVM.ps1 -ResourceGroupName $ResourceGroupName -Region $Region  -VirtualMachineSize $VirtualMachineSize  -DataDiskCount $DataDiskCount

The configuration example above has enough resources to run an OpenShift cluster.

Running the script will initially:

• Create a resource group

• Create a storage account

• copy the ASDK 2206 VHD image to the storage account

• Create the VM using the VHD image

• Create a Public IP for the VM

Note: As part of the provisioning process, the admin user account you specify gets changed to ‘Administrator’. I would Strongly recommend removing the Public IP associated with the VM and deploy Azure Bastion to protect your ASDK instance

Once the ASDK VM has been provisioned, connect to it (Bastion or RDP). The username you specified previously is ignored, so use ‘Administrator’ as the user and enter the password you defined.

Once connected, open a PowerShell window (as Administrator), and run the following as an example (I’m using ADFS as I’m simulating a disconnected environment)

C:\CloudDeployment\Setup\InstallAzureStackPOC.ps1 -TimeServer '129.6.15.28' -DNSForwarder '8.8.8.8' -UseADFS

You’ll then need to enter the AdminPassword when prompted, and then the script will do it’s magic (as long as the password is correct!) and take a number of hours to install.

The above recording shows the first few minutes of the script (sped-up! :) ).

After a few hours, the VM will reboot. If you want to check progress, you should use the following username to connect:

azurestackadmin@azurestack.local

Use the password you initially defined

Here’s some of the output you’ll see from PowerShell if you do connect as azurestackadmin (there’s still a few hours left to go!)

After 7hours 25 minutes, the install completed. You can determine this from the following log entry:

To prove that version 2206 has been installed, open the admin portal and check the properties for the region/instance.

As I used ADFS for this example, I had to login as cloudadmin@azurestack.local. If using AAD use the account you define when initially running the setup script.

View fullsize

Hope that helps if you want to deploy version 2206 as well as a simplified deployment tutorial.

Footnote: I tried using v5 series VM’s to deploy ASDK on, but it failed due to a network issue. I assume it is due to a different NIC/drive being used than the v3 series.