Trying to manually keep track of servers and versions is challenging. MAAS has a workload annotation feature that allows you to create name=value using command line calls using MASS cli. I wanted to explore creating a script to pull data from kubectl and create workload annotations for machines in a Kubernetes cluster.

For this exercise, I intend to execute the script from a MAAS rack controller, on which I have installed kubectl and PowerShell. In the lab, there are three microk8s clusters. The first job was to collect the config files from each cluster and combine them into a single config so I can use contexts from within the kubectl commands. Using the command ‘kubectl config view --raw > kn.conf’ I created one file for each k8s cluster on the region controller in directory ~/.kube/clusters.

mkdir ~/.kube/clusters
mv /path/k1.conf ~/.kube/clusters
mv /path/k2.conf ~/.kube/clusters

export KUBECONFIG=$(find ~/.kube/clusters -type f | sed ':a;N;s/\n/:/;ba')

kubectl config get-clusters

kubectl config view --flatten > ~/.kube/config

There are various methods to combine and clean up k8s cluster contexts. I had to clean up the file to create unique cluster names as well as user token names. I manually edited the ~/.kube/config combined file and tweaked these settings.

This is the script I created. It does require that you have a MAAS profile created already.

# tag-maasworkloadannotations.ps1

[CmdletBinding()]
param (
    $k8scontext ,
    $maasProfile
)
write-verbose "executing tag kubernetes workload annotation for MAAS"
$LastUpdated = Get-Date

# Kubectl contexts
Write-Verbose "Retrieving Kubernetes cluster details from context $k8scontext"
$kcontexts = kubectl config view -o json | convertfrom-json
Write-Verbose "$(($kcontexts.contexts | Measure-Object).count) kube contexts found"
$kcontext = $kcontexts.contexts | Where-Object {$_.name -eq $k8scontext}
$kversion = kubectl version --context $k8scontext -o json | convertfrom-json

# K8s nodes
Write-Verbose "Retrieving Kubernetes nodes from context $k8scontext"
$nodes = kubectl get nodes --context $k8scontext -o json | convertfrom-json
Write-Verbose "$(($nodes.items | Measure-Object).count) k8s nodes found"

# Maas machines
Write-Verbose "Retrieving machines from MAAS using profile $maasProfile"
$machines = maas $maasProfile machines read | convertfrom-json
Write-Verbose "$($machines.count) maas machines found"
$powerParams  = maas $maasProfile machines power-parameters  | convertfrom-json

# Build Annotations
Write-Verbose "Building workload annotation records"
$WorkloadAnnontations = @()
foreach ($node in $nodes.items) {

    $WorkloadAnnontation = @{}
    $WARecord = '' | select-object systemid, hostname, WA 

    $machine = $machines | Where-Object {$_.hostname -eq $node.metadata.name} 
    if ($machine -ne ""){
        $WARecord.systemid = $machine.system_id
        $WARecord.hostname = $machine.hostname

        #$WorkloadAnnontation.add("osImage",  $node.status.nodeInfo.osImage)
        #$WorkloadAnnontation.add("systemUUID",  $node.status.nodeInfo.systemUUID)
        #$WorkloadAnnontation.add("machineID",  $node.status.nodeInfo.machineID)

        $WorkloadAnnontation.add("k8scluster", $kcontext.context.cluster )
        $WorkloadAnnontation.add("buildDate", $kversion.serverVersion.buildDate)
        $WorkloadAnnontation.add("containerRuntimeVersion",   $node.status.nodeInfo.containerRuntimeVersion)
        $WorkloadAnnontation.add("kernelVersion",   $node.status.nodeInfo.kernelVersion)
        $WorkloadAnnontation.add("kubeProxyVersion",   $node.status.nodeInfo.kubeProxyVersion)
        $WorkloadAnnontation.add("kubeletVersion",   $node.status.nodeInfo.kubeletVersion)
        $WorkloadAnnontation.add("outofband",  $powerParams.$($machine.system_id).power_address)
        $WorkloadAnnontation.add("AnnotationUpdated",$LastUpdated)

        if ($node.metadata.labels.'node.kubernetes.io/microk8s-controlplane' -eq "microk8s-controlplane")
            {$WorkloadAnnontation.add("nodeType", 'Master')}
        if ($node.metadata.labels.'node.kubernetes.io/microk8s-worker' -eq "microk8s-worker")
            {$WorkloadAnnontation.add("nodeType", 'Worker')}

        $WARecord.wa = $WorkloadAnnontation
    }
    $WorkloadAnnontations += $WARecord
} 

# publish workload annotations
$i = 1
$c = $($WorkloadAnnontations.count)
Write-Verbose "Publishing $c workload annotation records"
foreach ($WA in $WorkloadAnnontations){
    $KeyValueData = ""
    foreach ($key in $WA.wa.keys) {
        $KeyValueData += "$($key)='$($wa.wa.($key))' "
    }
    Write-Verbose "[$i/$c] Building command for $($WA.hostname)"
    $execmd = "maas $maasProfile machine set-workload-annotations $($WA.systemID) $KeyValueData > /dev/null 2>&1"
    write-debug $execmd
    Invoke-Expression $execmd
    Write-Verbose "[$i/$c] Command executed for $($WA.hostname)" 
    $i++ 
}

$RunTime = New-TimeSpan -Start $LastUpdated -End (get-date) 
$ExecutionTime = "Execution time was {0} hours, {1} minutes, {2} seconds and {3} milliseconds." -f $RunTime.Hours,  $RunTime.Minutes,  $RunTime.Seconds,  $RunTime.Milliseconds 
write-verbose $ExecutionTime

Using the [CmdletBinding()] allows me to leverage verbose and debug options.

./tag-maasworkloadannotations.ps1 -k8scontext k8sdemo -maasprofile mquick -Verbose -debug

./tag-maasworkloadannotations.ps1 -k8scontext microk8s -maasprofile mquick -Verbose

This took 12 minutes but could probably create streamlined script with PowerShell jobs

which can allow combinations of filters on the workload annotations

There is a lot more possible here, this was helpful for me while consolidating microk8s clusters and making sure I wasn’t releasing machines that were in use in a cluster if I had tagged them incorrectly in MAAS.

External Reference

Using multiple kubeconfig files and how to merge to a single – Oueta

Creating MAAS workload annotations for Kubernetes using kubectl output

Simplifying Kubernetes deployments on ADFS Azure Stack systems