How to drain the node only after the new node is provisioned and in ready state?

[ghost]

Hi,

I have configured the node termination handler in Queue Processing mode with the Karpenter. But the node termination handler, cordons and drains the node immediately after receiving the spot interruption. But till the time the new node is provisioned all the pods are kept unscheduled. Since I am using the Karpenter it is not using the ASG.

What is the best way to manage the downtime while the node is provisioning?
Is there any way to cordon off the node as soon as Spot ITN is received and drain the pods when a new node is ready?

I really appreciate any help you can provide.

[snay2]

NTH does not have any visibility into new nodes joining the cluster, nor does it know which new node is logically intended to replace the node that received the ITN. So this may not be possible to do directly.

A few alternatives you could consider:

• The config value nodeTerminationGracePeriod lets you specify how long before the ITN deadline to start draining the node. You could experiment with lowering this value to see if you still have time to drain the pods before the node gets terminated. The default value, 120, basically means to start draining as soon as the ITN is received, since the ITN is issued 2 minutes before the actual termination deadline. This is potentially dangerous, so experiment in a safe environment first. See here:

  aws-node-termination-handler/config/helm/aws-node-termination-handler/values.yaml

  Lines 100 to 101 in 01f4844

  	# nodeTerminationGracePeriod specifies the period of time in seconds given to each NODE to terminate gracefully. Node draining will be scheduled based on this value to optimize the amount of compute time, but still safely drain the node before an event.
  	nodeTerminationGracePeriod: 120

• Add additional spare capacity to your cluster that can absorb pods during unpredictable events like this
• If this particular workload cannot tolerate downtime, consider running it on On-Demand nodes instead

[ghost]

Hello @snay2, thanks for the reply, I tried changing the nodeTerminationGracePeriod: 60 to drain the node before 60 seconds but it didn't work.

I updated the values.yaml file and redeployed the NTH but it still drains as soon as it receives spot ITN.

Please help me in this issue.