Navigate to your console URL provided by the below az command : 

az aro show \          
--name $CLUSTER \
--resource-group $RESOURCEGROUP \
--query "consoleProfile.url" -o tsv

where $CLUSTER and $RESOURCEGROUP are environment variables representing name of cluster and resoourcegroup, respectively.



Login using the credentials provided by below az command : 

az aro list-credentials \
--name $CLUSTER \  
--resource-group $RESOURCEGROUP



The dashboard provided by Red Hat has an user interface for managing entire cluster. Explore various options available in the left panel



We are interested in introducing spot instances for compute resources for our workloads. Let's dive-in further in  the Compute section.



Nodes section show us all the available cluster nodes, ofcourse 🙄. For a new cluster, by default we have 3 master and 3 worker nodes. Note that these are all on-demand virtual machines.



MachineSets are to Machines as ReplicaSets are to Pods. If you need more Machines or must scale them down, you change the Replicas field on the corresponding MachineSet to meet your compute need



Machines are nothing but the actual underlying infrastructure. Any changes to MachineSets are reflected in the corresponding Machines



To create a new Node with a spot instance we will eventually have to create a new Machine with the required configuration and that will be done via a new MachineSet. 
For ease, we will be picking up existing MachineSet YAML file and editing as per our need to create manifest for our MachineSet. Let's begin !!



It sounds scary but it is going to be a piece of cake as we walk through each steps, you'll see. 

First, go to the MachineSet section and click on the options icon



Click on "Edit MachineSet" to get the YAML for the MachineSet. 



Now, copy the entire manifest of the MachineSet



  uid: d44676fb-3f1f-4114-bd90-c212e4cb1998

Let's create our custom MachineSet by editing important details in the one we copied. Click on "Create MachineSet" option. There will be some default object definition present. Go-ahead and delete it and paste the copied content of the existing MachineSet.



The `managedFields` field in a YAML definition is automatically generated and managed by Kubernetes itself. It is not something that you manually create or modify. 
Let's get rid of entire section from our definition.



Now, the first change we need to do is give a unique name to our MachineSet. Change `metadata . name` as per your need and make sure you copy the name to keep it handy for further changes. 
In my case, I have changed the name (not so creatively 😄) from "spot-demo-cluster-fqc92-worker-eastus3" to "spot-demo-cluster-fqc92-worker-spot-instance"



Next, make the same change of name for "spec . selector . matchLabels . machine . openshift . io / cluster-api-machineset"



Also, the same change of name for "spec . template . metadata . labels . machine . openshift . io / cluster-api-machineset"



Further, add "spotVMOptions: {}" under "spec . template . spec . providerSpec . value". This is the actual attribute that will create spot-instance nodes for our cluster. 
Pay attention to the indentation as that is essential for YAML syntax to be correct and acceptable.



Optionally, to ensure that only targeted pods are scheduled on the spot nodes, we can add a toleration in a pod for " spot = 'true' " and taint the node by adding a taint under "spec . template . spec" as shown in the image.
Again, pay attention to the indentation. 



We are done with all the required changes now. Click "Create" to submit the definition and let Openshift do the magic ✨ ! 



After ~5mins, our custom MachineSet will have 1/1 machine ready.
Note that this is a spot instance and can be verified from the Azure Portal.



Moreover, the newly created machine has been added to our cluster as a new node ready to serve workload.



So, our spot node is ready. We can now safely scale down the existing worker node to zero machines and still have 3 worker nodes serving the workloads. 
Not-to-mention, the spot node will be much cheaper than the VM node.

Navigate to the MachineSets section and click on the menu option for the selected worker MachineSet we had copied the manifest from. 
You can choose any but this one will be recommended as our spot instance is create in the same availability zone as this machine.



Reduce the count to 0 and hit save.
Again, magic begins !



As expected, after some time machines for the updated MachineSet will be scaled down (0/0 in our case) releasing all the resources and can be scaled up easily anytime needed by just increasing the count.



The Node corresponding to that MachineSet will go in "NotReady" state and eventually will be deleted & removed from the cluster.



We will again have 6 machines of which one of the worker node will be a spot node.



Optionally, same process can be repeated for one more worker node. This will leave with one worker node running as a virtual machine which will always be serving load even if two of the spot nodes would be evicted by Azure.



Replacing a VM node with a Spot node on Azure Red Hat OpenShift